Digital watermark padding method, digital watermark padding device, digital watermark detecting method, digital watermark detecting device, and program

ABSTRACT

A digital watermark embedding method of the present invention includes: a step of sequentially obtaining each frame image of the moving image data and frame display time; a step of generating a watermark pattern using watermark information, the frame display time and watermark pattern switching information; a step of superimposing the watermark pattern onto the frame image, and combining watermark embedded frame images obtained by sequentially repeating the processes to generate watermark embedded moving image data. A digital watermark detection method includes a step of sequentially obtaining a frame image; a step of generating a difference image between the currently obtained frame image and a previously obtained frame image; and a step of performing digital watermark detection from the difference image to output digital watermark detection status, and when digital watermark detection process is continued, obtaining a new frame again to repeat the above processes.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. Ser. No. 11/666,311, filed Apr.26, 2007, which is a National Stage of PCT JP 06/315126 filed Jul. 31,2006, and claims the benefit of priority under 35 U.S.C. §119 ofJapanese applications and claims priority based on JP No. 2005-226755filed Aug. 4, 2005; JP No. 2005-226756 filed Aug. 4, 2005; and JP No.2005-275965 filed Sep. 22, 2005, and the entire contents of each areincorporated herein by reference.

TECHNICAL FIELD

The present invention relates to a digital watermark embedding techniqueand a digital watermark detection technique. More particularly, thepresent invention relates to a digital watermark embedding technique forembedding sub-information into image content such that it is notperceived by a human, and a digital watermark detection technique forreading the sub-information.

BACKGROUND ART

Today, the digital watermarking technique is used for a content copyright protection/management system, content related service providingsystem and the like.

For the purpose of content identification/management, copyrightprotection/management, providing related information and the like whendistributing content such as image, video and voice, there are methodsusing the digital watermark embedding technique for embedding otherinformation into the content such that it is not perceived.

For example, there is a using method for inputting a still image such asprinted material with a camera so as to obtain related information bydetecting digital watermark from the input image (refer to non-patentdocument 1, for example).

In addition, there is a method for continuously performing camera inputand digital watermark detection in real time to improve detectionperformance (refer to non-patent document 2, for example).

-   [Non-patent document 1] Nakamura, Katayama, Miyaji, Yamashita,    Yamamuro, “Digital watermark detection scheme for service mediation    using camera-equipped mobile-phone”, Forum on Information    Technology, FIT2003, N-020, September, 2003.-   [Non-patent document 2] Nakamura, Miyatake, Hayashi, Katayama,    Yamamuro, “Real time digital watermark detection scheme from camera    input image”, Forum on Information Technology, FIT2004, J-036,    September, 2004.

DISCLOSURE OF THE INVENTION Problem to be Solved by the Invention

However, there is no method for taking moving images such as images on aTV screen and performing digital watermark detection from the movingimages in real time, for example. If there is such method, informationrelated to the screen being broadcasted on the TV can be obtained, sothat various information services that can be linked up with the movingimage content in real time can be realized. However, there is a problemin that it is difficult to realize enough tolerability against noiseoccurring due to D/A conversion or A/D conversion associated with imagetaking by the camera. In addition, there is a problem in that it isdifficult to identify which part is a digital watermark detectionsubject in a captured frame image taken with camera.

The present invention is contrived in view of the above-mentionedpoints, and an object is to provide a technique that did not exist inthe past and that realizes taking moving images on a TV and the likewith a camera to detect digital watermark from the moving images in realtime. In addition, an object of the present invention is provide atechnique that realizes identifying a part from which the digitalwatermark can be detected in the captured frame image so as to be ableto detect digital watermark with reliability even under variousconditions of camera taking angles and background images.

Means for Solving the Problem

The object can be achieved by a digital watermark embedding method in adigital watermark embedding apparatus for embedding digital watermarkinto moving images, including:

a step of inputting moving image data including a frame image group,watermark information and watermark pattern switching informationspecifying temporal change of watermark patterns;

a frame image obtaining step of sequentially obtaining, by frame imageobtaining means, each frame image of the moving image data and framedisplay time that is display time of the frame image;

a watermark pattern generation step of generating, by watermark patterngeneration means, a watermark pattern using the watermark information,the frame display time and watermark pattern switching information;

a watermark pattern superimposing step of superimposing, by watermarkpattern superimposing means, the watermark pattern onto the frame image;and

a moving image data reconstruction step of combining watermark embeddedframe images obtained by sequentially repeating processes of the frameimage obtaining means, processes of the watermark pattern generationmeans and processes of the watermark pattern superimposing means togenerate watermark embedded moving image data.

In the watermark pattern generation step, timing for switching thewatermark pattern may be repeated in a constant period.

The watermark pattern generation step may includes:

a step of generating the watermark pattern corresponding to thewatermark information using the watermark information, the frame displaytime and the watermark pattern switching information; and

steps of generating a watermark pattern for phase determination, usingthe frame display time and the watermark pattern switching information,that is used for estimating temporal change status of the watermarkpattern when performing detection, and multiplexing the watermarkpattern for phase determination into the watermark pattern correspondingto the watermark information so as to obtain a pattern as the watermarkpattern.

The object can be also achieved by a digital watermark embedding methodin a digital watermark embedding apparatus for embedding digitalwatermark into moving images, including:

a step of inputting moving image data including a frame image group,watermark information and watermark pattern switching information thatis period information specifying phase change of watermark patterns;

a frame image obtaining step of sequentially obtaining, by frame imageobtaining means, each frame image of the moving image data and framedisplay time that is display time of the frame image;

a watermark pattern generation step of generating, by watermark patterngeneration means, a watermark pattern using the watermark information,the frame display time and watermark pattern switching information;

a watermark pattern superimposing step of superimposing, by watermarkpattern superimposing means, the watermark pattern onto the frame image;and

a moving image data reconstruction step of combining watermark embeddedframe images obtained by sequentially repeating steps from the frameimage obtaining step to the watermark pattern superimposing step togenerate watermark embedded moving image data.

In the digital watermark embedding method, the watermark patterngeneration step may include:

a basic watermark pattern generation step of generating a basicwatermark pattern using the watermark information; and

a step of adding phase change determined in the basic watermark patternto a next previous watermark pattern using the frame display time andthe watermark pattern switching information to generate a new watermarkpattern.

The watermark pattern generation step may include:

a basic watermark pattern generation step of generating a basicwatermark pattern using the watermark information;

a sign pattern generation step of generating a sign pattern based onpixel values of the basic watermark pattern;

a phase change value calculation step of obtaining a watermark patternswitching phase change value corresponding to time difference from anext previous frame using the frame display time and the watermarkpattern switching information;

watermark phase pattern generation steps of providing signs of eachelement of the sign pattern generated in the sign pattern generationstep to the watermark pattern switching phase change value to obtainphase differences from a next previous watermark phase pattern, andgenerating a watermark phase pattern for the current frame using thephase differences; and

a watermark phase pattern imaging step of generating the watermarkpattern based on the watermark phase pattern.

The watermark pattern generation step may include:

a step of increasing or decreasing amplitude of a corresponding pixelvalue in the watermark pattern based on each pixel value of the basicwatermark pattern when generating the watermark pattern from thewatermark phase pattern.

The watermark pattern generation step may include:

a step of associating phase difference change represented by the basicwatermark pattern with rotation amount in a coordinate system obtainedfrom image components so as to generate the watermark pattern based onnew component values obtained by rotating by the phase differencechange.

In the digital watermark embedding method, Cb-Cr components of an imagemay be used as the image components.

The watermark pattern generation step may include:

a step of dividing the watermark information into data blocks using thewatermark information, the frame display time and the watermark patternswitching information; and

a step of generating the watermark pattern based on a data block ID anddata block information of the data block ID which are determined fromthe frame display time and the watermark pattern switching information.

The watermark pattern generation step may include:

a step of dividing the watermark information into data blocks using thewatermark information, the frame display time and the watermark patternswitching information; and

a step of generating the watermark pattern such that the data block IDalso serves as watermark information for phase determination whengenerating the watermark pattern based on the data block ID and datablock information of the data block ID which are determined from theframe display time and the watermark pattern switching information.

The watermark superimposing step may include:

a step of changing scale of the watermark pattern to a size equal to orless than the frame image so as to superimpose the watermark pattern inthe inside of the frame image.

In the digital watermark embedding method, the watermark patterngeneration step may include:

a step of generating a basic watermark pattern using the watermarkinformation;

a step of adding a positioning pattern for extracting a detectionsubject region when performing digital watermark detection to the basicwatermark pattern; and

a step of generating the watermark pattern by changing the basicwatermark pattern using the frame display time and the watermark patternswitching information.

In the digital watermark embedding method, the watermark patterngeneration step may include steps of:

modulating the watermark information into the basic watermark patternusing existing two-dimensional code; and

generating the watermark pattern from the basic watermark pattern usingthe frame display time and the watermark pattern switching information.

The digital watermark embedding apparatus may include a plurality ofwatermark pattern generation means, and in the digital watermarkembedding step, each of the watermark pattern generation means generatesa different watermark pattern, and the watermark pattern superimposingmeans superimposes a plurality of watermark patterns onto the frameimage.

The watermark pattern superimposing step may include:

a step of amplifying amplitude of the watermark patter wherein thelarger movement included in the frame image is, the greater theamplitude of the watermark patter for the frame image is.

The watermark pattern superimposing step may include steps of:

storing a previously received frame image into storing means; and

generating a difference image between a currently received frame imageand the previous frame image to amplify the amplitude of the basicwatermark pattern based on pixel values of the difference image.

In the digital watermark embedding method, the watermark patternsuperimposing step may include a step of amplifying amplitude of thewhole of the watermark pattern.

Further, the watermark pattern superimposing step may include a step ofamplifying a pixel region of the watermark pattern corresponding to apixel region where movement is large in the frame image.

The object can be achieved by a digital watermark detection method in adigital watermark detection apparatus for detecting digital watermarkfrom moving images, including:

a moving image input step of sequentially obtaining a frame image bymoving image input means;

a difference image generation step of generating, by difference imagegeneration means, a difference image between the currently obtainedframe image and a previously obtained frame image; and

a digital watermark detection step of performing, by digital watermarkdetection means, digital watermark detection from the difference imageto output a digital watermark detection status,

wherein, in a case where digital watermark detection process iscontinued including a case where previous digital watermark detection isimpossible, the moving image input means obtains a new frame image againso that each of the above steps is repeated.

The digital watermark detection method may include:

obtaining a data block ID and data block information from the watermarkinformation obtained in the digital watermark detection step;

recording the detected data block information into informationcorresponding to the data block ID in a detected watermark informationbuffer; and

outputting information indicating that digital watermark detectionsucceeds when detection for every data block ID completes.

The moving image input step may include a step of sequentially obtainingthe frame image and frame display time that is display time of the frameimage, and

the digital watermark detection method may further include:

a determination step of determining detection necessity based on a timeinterval between the current frame display time and frame display timeof the previously obtained frame image using watermark pattern switchinginformation used when embedding digital watermark,

wherein, when it is determined that detection is unnecessary in thedetermination step, the moving image input means obtains a new frameimage again, and when it is determined that detection is necessary,processes after the difference image generation step are continued.

The digital watermark detection method may include a capture timingcontrol step of setting timing for capturing frames using the watermarkpattern switching information used when performing digital watermarkembedding,

wherein the moving image input means sequentially obtains the frameimage based on the capture timing.

The digital watermark detection method may include a step of settingdifference timing representing a time interval between frames forobtaining the difference image using the watermark pattern switchinginformation used when performing digital watermark embedding,

wherein the moving image input means sequentially obtains the frameimage and the frame display time, and the difference image generationmeans generates the difference image between the currently obtainedframe image and a frame image obtained a time before wherein the time isspecified by the difference timing.

In the digital watermark detection method, the digital watermarkdetection step may include steps of:

determining a phase of a watermark pattern embedded in the differenceimage; and

when the phase of the watermark pattern is reversed, performing bitreversal of detected watermark information and outputting the digitalwatermark detection status.

The digital watermark detection method may include:

adding the difference image to a difference image storing buffer in thedifference image generation step;

performing digital watermark detection from the difference image storingbuffer to detect watermark information and output the detection statusin the digital watermark detection step.

The digital watermark detection method may include:

a step of determining a phase of the watermark pattern embedded in thedifference image; and

a step of changing the difference image such that phases of alldifference images sequentially processed become the same,

wherein the difference image generation step and the digital watermarkdetection step are performed using the difference image obtained inthese steps.

The digital watermark detection step may include steps of:

storing a correlation value calculated when performing detection intostoring means each time when detection process is performed; and

determining whether detection succeeds using the stored correlationvalue.

The digital watermark detection step may include the steps of:

storing an absolute value of a correlation value calculated whenperforming detection into storing means each time when detection processis performed; and

determining whether detection succeeds using the stored absolute valueof the correlation value.

The digital watermark detection method may include:

a step of determining a phase of the watermark pattern in the differenceimage based on relationship between the watermark pattern switchinginformation used when performing embedding and the frame display timeobtained when obtaining the frame image.

The digital watermark detection method may include steps of:

detecting a bit value of a predetermined bit position in the watermarkinformation represented by the watermark pattern embedded in thedifference image; and

determining the phase of the watermark pattern in the difference imagebased on the bit value.

The digital watermark detection method may include:

a step of detecting watermark for phase determination so as to determinethe phase of the watermark pattern in the difference image.

The digital watermark detection method may include:

a step of determining the phase of the watermark pattern in thedifference image using polarity of plus or minus of a correlation valuecalculated in watermark detection process.

The digital watermark detection method may include steps of:

detecting data block information and a data block ID that are watermarkinformation by performing digital watermark detection from thedifference image; and

determining the phase of the watermark pattern embedded in thedifference image using polarity of plus or minus of a correlation valueobtained when detecting the data block ID.

The difference image generation step may include the steps of:

generating the difference image between the currently obtained frameimage and a previously obtained frame image;

obtaining a phase value and an amplitude value based on a particularcomponent in the difference image;

determining a group, among a plurality of given groups, to which thephase value belongs; and

correcting the difference image based on the determination result andthe amplitude value to output the difference image.

In the digital watermark detection method, the difference imagegeneration step may include steps of generating a difference image (A)between the currently obtained frame image and a previously obtainedframe image read from image storing means, and storing the currentlyobtained frame image into the image storing means, and

the digital watermark detection method may include phase differencemeasurement steps of:

generating a phase pattern based on the currently obtained differenceimage (A);

measuring a phase difference between the currently obtained phasepattern and a previously obtained phase pattern read from phase patternstoring means;

generating a difference image (B) based on the phase difference; and

storing the currently obtained phase pattern in the phase patternstoring means, and

the digital watermark detection step may include a step of performingdigital watermark detection from the difference image (B) to output thedigital watermark detection status.

The phase difference measurement step may include steps of:

when generating the phase pattern from the difference image (A),obtaining an amplitude value with a phase value as an element value ofthe phase pattern;

measuring a phase difference between the currently obtained phasepattern and a previously obtained phase pattern to determine a signbased on the phase difference; and

generating the difference image (B) based on amplitude values of thecurrently obtained phase pattern and the previously obtained phasepattern, and the sign.

The phase difference measurement step may include steps of:

when generating the phase pattern from the difference image (A),obtaining an amplitude as a value monotonically increasing with respectto amplitude values of the currently obtained phase pattern and thepreviously obtained phase pattern; and

obtaining the difference image (B) based on a value obtained bycombining the amplitude and the sign.

In the digital watermark detection method, the particular component inthe difference image in the difference image generation step may includea plurality of components.

The digital watermark detection method may include:

when obtaining the phase pattern from the difference image (A) in thephase difference measurement step, using a phase and an amplitude in acoordinate system obtained from a plurality of components in thedifference image (A).

The particular component in the difference image may be a Cb componentand a Cr component.

The digital watermark detection method may further include a step ofadding the difference image (B) into a difference image storing buffer,and

the digital watermark detection step may include a step of performingdigital watermark detection from the difference image storing buffer todetect watermark information and output the detection status.

The digital watermark detection method may include steps of:

extracting a detection subject region in a current frame image ofsequentially obtained frame images; and

correcting distortion and normalizing the size to generate a detectionsubject region image, and

in the difference image generation step, generating a difference imagebetween a currently obtained detection subject region image and apreviously obtained detection subject region image.

The digital watermark detection method may include steps of:

extracting a detection subject region from the sequentially obtaineddifference image to generate a detection subject region image, and

in the difference image generation step, performing digital watermarkdetection from the detection subject region image to output thedetection status.

The digital watermark detection method may include:

a feature region extraction step of, after sequentially obtaining theframe image, extracting a feature region from the frame image, andperforming distortion correction and size normalization to generate afeature region image;

a difference image generation step of generating a difference imagebetween a currently obtained feature region image and a previouslyobtained feature region image;

a detection subject region extraction step of extracting a detectionsubject region from the difference image to generate a detection subjectregion image; and

a digital watermark detection step of performing digital watermarkdetection from the detection subject region.

The detection subject region extraction step may include a step of, whenextracting the detection subject region from the difference image,changing pixel values of the difference image into absolute values andsearching for the detection subject region.

The detection subject region extraction step may include a step of, whenextracting the detection subject region from the difference image,searching for the detection subject region using a positioning patternadded to the basic watermark pattern used when performing embedding.

The detection subject region extraction step may include a step of:

searching a neighborhood region of a detection subject region where thedetection status is good in past digital watermark detection trial toperform current detection subject region extraction process.

The detection subject region extraction step may include a step of:

determining the phase of the watermark pattern in the difference imageusing the positioning pattern.

The detection subject region extraction step may include steps of:

adding and storing the difference image into storing means; and

searching for and extracting the detection subject region from thestored difference image.

The detection subject region extraction step may include a step of:

when adding and storing the difference image into the storing means,aligning the phase of the watermark pattern in the difference image toperform the adding and storing.

The object can be also achieved by a digital watermark detection methodin a digital watermark detection apparatus for detecting digitalwatermark from moving images, including:

a frame image obtaining step of sequentially obtaining a frame image bymoving image input means;

a feature region extraction step of extracting, by feature regionextraction means, a feature region in the frame image to obtain afeature region image;

a difference image generation step of generating, by difference imagegeneration means, a difference image between a currently obtainedfeature region image and a previously obtained feature region image readfrom a feature region image buffer to obtain a difference image (A), andstoring the currently obtained feature region image into the featureregion image buffer;

a phase difference measurement step, performed by phase differencemeasurement means, of generating a phase pattern based on the currentlyobtained difference image (A), measuring a phase difference between thecurrently obtained phase pattern and a previously obtained phase patternread from a phase pattern buffer, generating a difference image (B)based on the phase difference and amplitude, and storing the currentlyobtained difference image (A) into the phase pattern buffer;

a detection subject region extraction step, performed by detectionsubject region extraction means, of performing adding and storing forpixel values of the difference image (B) for each pixel, extracting adetection subject region from a difference image (B) storing bufferobtained by the adding and storing to obtain a detection subject regionimage;

a digital watermark detection step of performing, by digital watermarkdetection means, digital watermark detection from the detection subjectregion image to output digital watermark detection status; and

a step of obtaining, by the moving image input means, a new frame imageagain so as to repeat each of the above steps in a case where digitalwatermark detection process is continued including a case where digitalwatermark detection is impossible.

The detection subject region extraction step may include the steps of:

when extracting the detection subject region from the difference image(B) obtained by adding and storing, extracting the detection subjectregion from pixel values that are converted to absolute values in thedifference image (B) storing buffer; and

obtaining an image of the detection subject region in the differenceimage (B) storing buffer before being converted to the absolute valuesas the detection subject region image.

The digital watermark detection method may include:

extracting a plurality of detection subject regions in the detectionsubject extraction step; and

performing digital watermark detection trial for each of the pluralityof detection subject regions to output the detection result in thedigital watermark detection step.

The digital watermark detection step may include a step of detectingwatermark information using decoding processes for existingtwo-dimensional code when detecting the watermark information.

In the digital watermark detection method may include a step of:

performing filtering process on the difference image generated bysubtraction process to output a difference image to which the filteringprocess is performed. A non-linear filter may be used as the filteringprocess.

In the digital watermark detection method, the moving images to be inputmay be video signals that are taken by a camera and captured.

The digital watermark detection method may include a step, performed byexternal output means for outputting a screen or sound, of performingfeedback output according to any one or more of detection subject regionextraction status, digital watermark detection status and data blockdetection status.

The digital watermark detection method may include:

further generating detection subject region information indicating asize and a position of the detection subject region in the detectionsubject region extraction step; and

a zooming process step of setting a zooming parameter for changing thedetection subject region to a given pixel size based on the detectionsubject region information, and performing zooming process by supplyingthe zooming parameter to the moving image input means.

The zooming process step may include steps of:

when the detection status in the digital watermark detection step isgood, setting the zooming parameter for changing the detection subjectregion to a given pixel size based on the detection subject regioninformation, and performing the zooming process by supplying the zoomingparameter to the moving image input means.

The moving images to be input may captured by taking a state displayedon a display by a camera in real time, and the digital watermarkdetection method may include a step of obtaining information related tothe displayed moving images based on watermark information obtained bydigital watermark detection.

In addition, according to the present invention, a digital watermarkembedding apparatus, a digital watermark detection apparatus, a programand a recording medium storing the program that are suitable forcarrying out the method are provided.

Effect of the Invention

According to the present invention, a technique that realizes takingmoving images on a TV and the like with a camera to detect digitalwatermark from the moving images in real time can be provided. Inaddition, a technique that realizes identifying a part from which thedigital watermark can be detected in the captured frame image so as tobe able to detect digital watermark with reliability even under variousconditions of camera taking angles and background images can beprovided.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a flowchart of a digital watermark embedding method showingan outline A of an embodiment;

FIG. 1B is a flowchart of a digital watermark detection method showingthe outline A of the embodiment;

FIG. 2A is a block diagram of a digital watermark embedding apparatusshowing the outline A of the embodiment;

FIG. 2B is a block diagram of a digital watermark detection apparatusshowing the outline A of the embodiment;

FIG. 3A is a flowchart of a digital watermark embedding method showingan outline B of an embodiment;

FIG. 3B is a flowchart of a digital watermark detection method showingthe outline B of the embodiment;

FIG. 4A is a block diagram of a digital watermark embedding apparatusshowing the outline B of the embodiment;

FIG. 4B is a block diagram of a digital watermark detection apparatusshowing the outline B of the embodiment;

FIG. 5A is a flowchart of a digital watermark embedding method showingan outline C of an embodiment;

FIG. 5B is a flowchart of a digital watermark detection method showingthe outline C of the embodiment;

FIG. 6A is a block diagram of a digital watermark embedding apparatusshowing the outline C of the embodiment;

FIG. 6B is a block diagram of a digital watermark detection apparatusshowing the outline C of the embodiment;

FIG. 7 is a block diagram a digital watermark embedding apparatus of afirst embodiment of the present invention;

FIG. 8 is a flowchart of operation of the digital watermark embeddingapparatus of the first embodiment of the present invention;

FIG. 9 is a figure for explaining processes of a frame image obtainingunit in the first embodiment of the present invention;

FIG. 10 is a block diagram of a watermark pattern generation unit in thefirst embodiment of the present invention;

FIG. 11 is a flowchart of operation of the watermark pattern generationunit in the first embodiment of the present invention;

FIG. 12 is a figure (1) for explaining an example of processes of thebasic watermark pattern generation unit in the first embodiment of thepresent invention;

FIG. 13 is a figure (2) for explaining an example of processes of thebasic watermark pattern generation unit in the first embodiment of thepresent invention;

FIG. 14A is a figure for explaining an example of watermark patternswitching information in the first embodiment of the present invention;

FIG. 14B is a figure for explaining an example of watermark patternswitching information in the first embodiment of the present invention;

FIG. 14C is a figure for explaining an example of watermark patternswitching information in the first embodiment of the present invention;

FIG. 15 is a figure (1) for explaining processes of the watermarkpattern switching unit in the first embodiment of the present invention;

FIG. 16 is a figure (2) for explaining processes of the watermarkpattern switching unit in the first embodiment of the present invention;

FIG. 17 is a figure for explaining processes of the watermark patternsuperimposing unit in the first embodiment of the present invention;

FIG. 18 is a figure for explaining a using scene of the presentinvention;

FIG. 19 is a block diagram of the digital watermark detection apparatusin the first embodiment of the present invention;

FIG. 20 is a flowchart of operation of the digital watermark detectionapparatus in the first embodiment of the present invention;

FIG. 21 is a figure for explaining processes of the detection subjectregion extraction unit in the first embodiment of the present invention;

FIG. 22 is a figure for explaining processes of the difference imagegeneration unit in the first embodiment of the present invention;

FIG. 23 is a figure for explaining an example of processes of thedigital watermark detection unit in the first embodiment of the presentinvention;

FIG. 24 is a figure for explaining effects of the first embodiment ofthe present invention;

FIG. 25 is a block diagram of the digital watermark detection apparatusin the second embodiment of the present invention;

FIG. 26 is a flowchart of operation of the digital watermark detectionapparatus in the second embodiment of the present invention;

FIG. 27 is a figure for explaining processes of the detection necessitydetermination unit in the second embodiment of the present invention;

FIG. 28A is a figure for explaining effects of the second embodiment ofthe present invention;

FIG. 28B is a figure for explaining effects of the second embodiment ofthe present invention;

FIG. 29 is a block diagram of the digital watermark detection apparatusin the third embodiment of the present invention;

FIG. 30 is a flowchart of operation of the digital watermark detectionapparatus in the third embodiment of the present invention;

FIG. 31 is a figure (1) for explaining processes of the capture timingcontrol unit in the third embodiment of the present invention;

FIG. 32 is a figure (2) for explaining processes of the capture timingcontrol unit in the third embodiment of the present invention;

FIG. 33A is a figure (1) for explaining effects of the third embodimentof the present invention;

FIG. 33B is a figure (2) for explaining effects of the third embodimentof the present invention;

FIG. 34A is a figure for explaining processes of the basic watermarkpattern in the fourth embodiment of the present invention;

FIG. 34B is a figure for explaining processes of the basic watermarkpattern in the fourth embodiment of the present invention;

FIG. 35 is a flowchart of operation of the digital watermark detectionunit in the fourth embodiment of the present invention;

FIG. 36 is a figure (1) for explaining effects of the fourth embodimentof the present invention;

FIG. 37 is a figure (2) for explaining effects of the fourth embodimentof the present invention;

FIG. 38 is a figure (3) for explaining effects of the fourth embodimentof the present invention;

FIG. 39 is a figure for explaining an example of processes of the basicwatermark pattern generation unit in the fifth embodiment of the presentinvention;

FIG. 40 is a flowchart of operation of the digital watermark detectionunit in the fifth embodiment of the present invention;

FIG. 41 is a block diagram of the digital watermark detection unit inthe sixth embodiment of the present invention;

FIG. 42 is a flowchart of operation of the digital watermark detectionunit in the sixth embodiment of the present invention;

FIG. 43 is a figure for explaining processes of the difference imagephase determination unit in the sixth embodiment of the presentinvention;

FIG. 44 is a figure for explaining a difference image storing buffer ofthe sixth embodiment of the present invention;

FIG. 45 is a block diagram of the digital watermark detection unit inthe seventh embodiment of the present invention;

FIG. 46 is a flowchart of operation of the digital watermark detectionunit in the seventh embodiment of the present invention;

FIG. 47 is a figure for explaining contents of processes on thecorrelation value buffer in the seventh embodiment of the presentinvention;

FIG. 48A is a figure for explaining contents of processes on thedetection availability determination unit in the seventh embodiment ofthe present invention;

FIG. 48B is a figure for explaining contents of processes on thedetection availability determination unit in the seventh embodiment ofthe present invention;

FIG. 49 is a block diagram of the watermark pattern generation unit inthe eighth embodiment of the present invention;

FIG. 50 is a flowchart of operation of the watermark pattern generationunit in the eighth embodiment of the present invention;

FIG. 51 is a figure for explaining processes of the watermarkinformation dividing unit in the eighth embodiment of the presentinvention;

FIG. 52A is a figure for explaining processes of the basic watermarkpattern generation unit in the eighth embodiment of the presentinvention;

FIG. 52B is a figure for explaining processes of the basic watermarkpattern generation unit in the eighth embodiment of the presentinvention;

FIG. 53 is a block diagram of the digital watermark detection apparatusin the eighth embodiment of the present invention;

FIG. 54 is a flowchart of operation of the digital watermark detectionapparatus in the eighth embodiment of the present invention;

FIG. 55 is a block diagram of the detected data block storing unit inthe eighth embodiment of the present invention;

FIG. 56 is a flowchart of operation of the detected data block storingunit in the eight embodiment of the present invention;

FIG. 57 is a figure (1) for explaining processes of the detected datablock storing unit in the eight embodiment of the present invention;

FIG. 58 is a figure (2) for explaining processes of the detected datablock storing unit in the eight embodiment of the present invention;

FIG. 59 is a figure for explaining an example of processes of the basicwatermark pattern generation unit in the ninth embodiment of the presentinvention;

FIG. 60 is a flowchart of operation of the digital watermark detectionunit in the ninth embodiment of the present invention;

FIG. 61 is a block diagram of the digital watermark detection unit inthe tenth embodiment of the present invention;

FIG. 62 is a flowchart of operation of the digital watermark detectionunit in the tenth embodiment of the present invention;

FIG. 63 is a block diagram of the digital watermark detection unit inthe eleventh embodiment of the present invention;

FIG. 64 is a flowchart showing operation of the digital watermarkdetection unit in the eleventh embodiment of the present invention;

FIG. 65 is a block diagram of a digital watermark detection apparatus ofthe twelfth embodiment of the present invention;

FIG. 66 is a flowchart of operation of the digital watermark detectionapparatus in the twelfth embodiment of the present invention;

FIG. 67 shows an example (1) of feedback output of the digital watermarkdetection apparatus in the twelfth embodiment of the present invention;

FIG. 68 shows an example (2) of feedback output of the digital watermarkdetection apparatus in the twelfth embodiment of the present invention;

FIG. 69 shows an example of feedback output of the digital watermarkdetection apparatus in the twelfth embodiment of the present invention;

FIG. 70 is a figure for explaining processes of the watermark patternsuperimposing unit in the thirteenth embodiment of the presentinvention;

FIG. 71 is a block diagram of the digital watermark detection apparatusin the thirteenth embodiment of the present invention;

FIG. 72 is a flowchart of operation of the digital watermark detectionapparatus in the thirteenth embodiment of the present invention;

FIG. 73 is a figure for explaining processes of the difference imagegeneration unit in the thirteenth embodiment of the present invention;

FIG. 74 is a figure for explaining processes of the detection subjectregion extraction unit in the thirteenth embodiment of the presentinvention;

FIG. 75 is a figure (1) for explaining effects of thirteenth embodimentof the present invention;

FIG. 76 is a figure (2) for explaining effects of thirteenth embodimentof the present invention;

FIG. 77 is a figure (3) for explaining effects of thirteenth embodimentof the present invention;

FIG. 78 is a figure (4) for explaining effects of thirteenth embodimentof the present invention;

FIG. 79 is a figure for explaining processes of the basic watermarkpattern generation unit of the fourteenth embodiment of the presetinvention;

FIG. 80 is a figure for explaining processes of the detection subjectregion extraction unit in the fourteenth embodiment of the presentinvention;

FIG. 81A is a figure for explaining effects of the fourteenth embodimentof the present invention;

FIG. 81B is a figure for explaining effects of the fourteenth embodimentof the present invention;

FIG. 82 is a figure for explaining processes of the basic watermarkpattern generation unit of the fifteenth embodiment of the presetinvention;

FIG. 83 is a figure for explaining processes of the detection subjectregion extraction unit in the fifteenth embodiment of the presentinvention;

FIG. 84 is a block diagram of the digital watermark detection apparatusin the sixteenth embodiment of the present invention;

FIG. 85 is a flowchart of operation of the digital watermark detectionapparatus in the sixteenth embodiment of the present invention;

FIG. 86 is a figure (1) for explaining processes of the feature regionextraction unit in the sixteenth embodiment of the present invention;

FIG. 87 is a figure (2) for explaining processes of the feature regionextraction unit in the sixteenth embodiment of the present invention;

FIG. 88 is a figure (3) for explaining processes of the feature regionextraction unit in the sixteenth embodiment of the present invention;

FIG. 89 is a figure (4) for explaining processes of the feature regionextraction unit in the sixteenth embodiment of the present invention;

FIG. 90 is a figure (5) for explaining processes of the feature regionextraction unit in the sixteenth embodiment of the present invention;

FIG. 91 is a figure (1) for explaining processes of the difference imagegeneration unit in the sixteenth embodiment of the present invention;

FIG. 92 is a figure (2) for explaining processes of the difference imagegeneration unit in the sixteenth embodiment of the present invention;

FIG. 93 is a figure (3) for explaining processes of the difference imagegeneration unit in the sixteenth embodiment of the present invention;

FIG. 94 is a figure for explaining processes of the feature regionextraction unit in the seventeenth embodiment of the present invention;

FIG. 95 is a figure for explaining processes of the feature regionextraction unit in the seventeenth embodiment of the present invention;

FIG. 96 is a figure (1) for explaining processes of the detectionsubject region extraction unit in the eighteenth embodiment of thepresent invention;

FIG. 97 is a figure (2) for explaining processes of the detectionsubject region extraction unit in the eighteenth embodiment of thepresent invention;

FIG. 98 is a figure (3) for explaining processes of the detectionsubject region extraction unit in the eighteenth embodiment of thepresent invention;

FIG. 99 is a figure (4) for explaining processes of the detectionsubject region extraction unit in the eighteenth embodiment of thepresent invention;

FIG. 100 is a block diagram of the digital watermark detection apparatusin the nineteenth embodiment of the present invention;

FIG. 101 is a flowchart showing operation of the digital watermarkdetection apparatus in the nineteenth embodiment of the presentinvention;

FIG. 102 is a figure (1) for explaining processes of the detectionsubject region extraction unit in the nineteenth embodiment of thepresent invention;

FIG. 103 is a figure (2) for explaining processes of the detectionsubject region extraction unit in the nineteenth embodiment of thepresent invention;

FIG. 104 is a block diagram of the digital watermark detection apparatusin the nineteenth embodiment of the present invention;

FIG. 105 is a flowchart of operation of the digital watermark detectionapparatus in the nineteenth embodiment of the present invention;

FIG. 106 is a figure (1) for explaining the detection subject regionextraction unit in the nineteenth embodiment of the present invention;

FIG. 107 is a figure (2) for explaining the difference image generationunit in the nineteenth embodiment of the present invention;

FIG. 108 is a figure (1) for explaining the processes of the watermarkpattern superimposing unit in the twentieth embodiment of the presentinvention;

FIG. 109 is a figure (2) for explaining the processes of the watermarkpattern superimposing unit in the twentieth embodiment of the presentinvention;

FIG. 110 is a figure for explaining processes of the difference imagegeneration unit in the twentieth embodiment of the present invention;

FIG. 111 is a figure (1) for explaining processes of the detectionsubject region extraction unit in the twentieth embodiment of thepresent invention;

FIG. 112 is a figure (1) for explaining processes of the detectionsubject region extraction unit in the twentieth embodiment of thepresent invention;

FIG. 113 shows an example (1) of feedback output of the digitalwatermark detection apparatus in the twentieth embodiment of the presentinvention;

FIG. 114 shows an example (2) of feedback output of the digitalwatermark detection apparatus in the twentieth embodiment of the presentinvention;

FIG. 115 shows an example (3) of feedback output of the digitalwatermark detection apparatus in the twentieth embodiment of the presentinvention;

FIG. 116 shows an example (4) of feedback output of the digitalwatermark detection apparatus in the twentieth embodiment of the presentinvention;

FIG. 117 is a figure for explaining an example of watermark patternswitching information in the twenty first embodiment of the presentinvention;

FIG. 118 is a figure for explaining processes of the watermark patternswitching unit in the twenty first embodiment of the present invention;

FIG. 119 is a figure for explaining processes of the watermark patternswitching unit in the twenty first embodiment of the present invention;

FIG. 120 is a figure for explaining processes of the difference imagegeneration unit in the twenty first embodiment of the present invention;

FIG. 121 is a figure (1) for explaining principle of the twenty firstembodiment of the present invention;

FIG. 122 is a figure (2) for explaining principle of the twenty firstembodiment of the present invention;

FIG. 123 is a figure (3) for explaining principle of the twenty firstembodiment of the present invention;

FIG. 124 is a figure for explaining effects of the twenty firstembodiment of the present invention;

FIG. 125 is a figure for explaining processes of the difference imagegeneration unit in the twenty second embodiment of the presentinvention;

FIG. 126A is a figure for explaining an example of filtering process inthe twenty second embodiment of the present invention;

FIG. 126B is a figure for explaining an example of filtering process inthe twenty second embodiment of the present invention;

FIG. 127 is a figure for explaining processes of the watermark patternsuperimposing unit in the twenty third embodiment of the presentembodiment;

FIG. 128 is a figure for explaining amplitude adjustment for thewatermark pattern in the twenty third embodiment of the presentinvention;

FIG. 129 is a block diagram of a digital watermark detection apparatusin the twenty fourth embodiment of the present invention;

FIG. 130 is a flowchart of operation of the digital watermark detectionapparatus in the twenty fourth embodiment of the present invention;

FIG. 131A is a figure for explaining effects of the twenty fourthembodiment of the present invention;

FIG. 131B is a figure for explaining effects of the twenty fourthembodiment of the present invention;

FIG. 132 is a figure for explaining an example of watermark patternswitching information in the twenty fifth embodiment of the presentinvention;

FIG. 133 is a figure (1) for explaining processes of the watermarkpattern switching unit in the twenty fifth embodiment of the presentinvention;

FIG. 134 is a figure (2) for explaining processes of the watermarkpattern switching unit in the twenty fifth embodiment of the presentinvention;

FIG. 135 is a block diagram of the digital watermark detection apparatusin the twenty fifth embodiment of the present invention;

FIG. 136 is a flowchart of operation of the digital watermark detectionapparatus in the twenty fifth embodiment of the present invention;

FIG. 137 is a figure for explaining processes of the difference imagegeneration unit in the twenty fifth embodiment of the present invention;

FIG. 138 is a figure (1) for explaining contents of processes of thephase difference measurement unit in the twenty fifth embodiment of thepresent invention;

FIG. 139 is a figure (2) for explaining contents of processes of thephase difference measurement unit in the twenty fifth embodiment of thepresent invention;

FIG. 140A is a figure (3) for explaining contents of processes of thephase difference measurement unit in the twenty fifth embodiment of thepresent invention;

FIG. 140B is a figure (3) for explaining contents of processes of thephase difference measurement unit in the twenty fifth embodiment of thepresent invention;

FIG. 141 is a figure (1) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 142 is a figure (2) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 143 is a figure (3) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 144 is a figure (4) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 145 is a figure (5) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 146 is a figure (6) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 147 is a figure (7) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 148 is a figure (8) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 149 is a figure (9) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 150 is a figure (10) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 151 is a figure (11) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 152 is a figure (12) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 153 is a figure (13) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 154 is a figure (14) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 155 is a figure (15) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 156 is a figure (16) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 157 is a figure (17) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 158 is a figure (18) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 159 is a figure (19) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 160 is a figure (20) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 161 is a figure (21) for explaining effects of the twenty fifthembodiment of the present invention;

FIG. 162 is a block diagram of the digital watermark embedding apparatusin the twenty sixth embodiment of the present invention;

FIG. 163 is a flowchart of operation of the digital watermark embeddingapparatus of the twenty sixth embodiment of the present invention;

FIG. 164 is a figure for explaining processes of the frame imageobtaining unit in the twenty sixth embodiment of the present invention;

FIG. 165 is a block diagram of the watermark pattern generation unit inthe twenty sixth embodiment of the present invention;

FIG. 166 is a flowchart of operation of the watermark pattern generationunit in the twenty sixth embodiment of the present invention;

FIG. 167 is a figure for explaining an example of processes of the basicwatermark pattern generation unit in the twenty sixth embodiment of thepresent invention;

FIG. 168 is a block diagram of the watermark pattern switching unit inthe twenty sixth embodiment of the present invention;

FIG. 169 is a flowchart of the operation of the watermark patternswitching unit in the twenty sixth embodiment of the present invention;

FIG. 170 is a figure for explaining an example of the watermark patternswitching information in the twenty sixth embodiment of the presentinvention;

FIG. 171 is a figure for explaining processes of the watermark patternswitching unit in the twenty sixth embodiment of the present invention;

FIG. 172 is a figure for explaining processes of the watermark phasepattern imaging unit in the twenty sixth embodiment of the presentinvention;

FIG. 173 is a figure for explaining processes of the watermark patternsuperimposing unit in the twenty sixth embodiment of the presentinvention;

FIG. 174 is a block diagram of the digital watermark detection apparatusin the twenty sixth embodiment of the present invention;

FIG. 175 is a flowchart of operation of the digital watermark detectionapparatus in the twenty sixth embodiment of the present invention;

FIG. 176 is a figure for explaining processes of the feature regionextraction unit in the twenty sixth embodiment of the present invention;

FIG. 177 is a figure for explaining processes of the difference imagegeneration unit in the twenty sixth embodiment of the present invention;

FIG. 178 is a figure (1) for explaining process contents of the phasedifference calculation unit in the twenty sixth embodiment of thepresent invention;

FIG. 179 is a figure (2) for explaining process contents of the phasedifference calculation unit in the twenty sixth embodiment of thepresent invention;

FIG. 180A is a figure (3) for explaining process contents of the phasedifference calculation unit in the twenty sixth embodiment of thepresent invention;

FIG. 180B is a figure (3) for explaining process contents of the phasedifference calculation unit in the twenty sixth embodiment of thepresent invention;

FIG. 181 is a figure for explaining the difference image (B) storingbuffer in the twenty sixth embodiment of the present invention;

FIG. 182 is a figure for explaining processes of the detection subjectregion extraction unit in the twenty sixth embodiment of the presentinvention;

FIG. 183 is a figure for explaining processes of the digital watermarkdetection unit in the twenty sixth embodiment of the present invention.

DESCRIPTION OF REFERENCE SIGNS

-   100 digital watermark embedding apparatus-   110 frame image obtaining unit-   120 watermark pattern generation unit-   121 basic watermark pattern generation unit-   122 watermark pattern switching unit-   123 watermark information dividing unit-   130 watermark pattern superimposing unit-   140 moving image data reconstruction unit-   200 digital watermark detection apparatus-   210 moving image input unit-   220 detection subject region extraction unit-   230 difference image generation unit-   240 digital watermark detection unit-   241 difference image phase determination unit-   242 watermark information detection unit-   243 correlation calculation unit-   244 detection availability determination unit-   246 data block ID detection unit-   247 data block information detection unit-   248 difference image storing buffer for data block ID==n-   249 correlation calculation unit-   250 detection subject region image buffer-   260 detection necessity determination unit-   265 feedback output unit-   270 capture timing control unit-   281 detected data block buffering unit-   282 detection complete check unit-   283 detected watermark information buffer-   290 feature region extraction unit-   301 frame image buffer-   302 feature region image buffer-   303 difference frame image buffer-   304 feature region difference image buffer-   310 zooming process unit-   360 phase difference measurement unit-   370 phase pattern buffer-   390 difference image (B) storing buffer-   1221 sign pattern generation unit-   1222 watermark pattern switching phase change value calculation unit-   1223 watermark phase difference pattern generation unit-   1224 watermark phase pattern generation unit-   1225 watermark phase pattern imaging unit

PREFERRED EMBODIMENTS FOR CARRYING OUT THE INVENTION

In the following, embodiments of the present invention are describedwith reference to figures.

Outline of the Embodiments

First, outlines of the embodiments are described. FIGS. 1A-2B show anembodiment outline A.

FIG. 1A shows a flowchart of a digital watermark embedding method of theembodiment outline A. This method is a digital watermark embeddingmethod in an digital watermark embedding apparatus for embedding digitalwatermark into moving images. In the method, moving image data includinga frame image group, watermark information, and watermark patternswitching information are supplied to the digital watermark embeddingapparatus (step A1). Next, the digital watermark embedding apparatussequentially obtains each frame image of the moving image data and itsframe display time (step A2), generates a watermark pattern using thewatermark information, the frame display time and the watermark patternswitching information (step A3) and superimposes the watermark patteronto the frame image (step A4). Then, the watermark embedding apparatuscombines watermark embedded frame images obtained by sequentiallyrepeating the step 2-step 4 so as to generate watermark embedded movingimage data (step A5).

FIG. 1B shows a flowchart of a digital watermark detection method of theembodiment outline A. This method is a digital watermark detectionmethod in a digital watermark detection apparatus for detecting digitalwatermark from the moving images. In the method, a frame image issequentially obtained, first (step A6). Next, the digital watermarkdetection apparatus generates a difference image between a currentlyobtained frame image and a previously obtained frame image (step A7) andperforms digital watermark detection from the difference image to outputdigital watermark detection status (step A8). Then, in cases where thedigital watermark detection process is continued including a case wheredigital watermark detection was impossible, the digital watermarkdetection apparatus obtains a new frame image again with a moving imageinput means and repeats the above-mentioned process (step A9).

FIG. 2A shows a block diagram of the digital watermark embeddingapparatus of the embodiment outline A. The digital watermark apparatusincludes frame image obtaining means A110 configured to sequentiallyobtain each frame image of moving image data including a supplied frameimage group and the frame display time, watermark pattern generationmeans A120 configured to generate a watermark pattern using the suppliedwatermark information, the frame display time and the supplied patternswitching information, watermark pattern superimposing means A130configured to superimpose the watermark pattern onto the frame image,and moving image data reconstruction means A140 configured to combinethe watermark embedded frame images that are obtained by sequentiallyrepeating processes from the frame image obtaining means A110 to thewatermark pattern superimposing means so as to generate watermarkembedded moving image data.

FIG. 2B shows a block diagram of the digital watermark detectionapparatus of the embodiment outline A. This digital watermark detectionapparatus includes moving image input means A210 configured tosequentially obtain a frame image, difference image generation meansA230 configured to generate a difference image between a currentlyobtained frame image and a previously obtained frame image, and digitalwatermark detection means A240 configured to perform digital watermarkdetection from the difference image to output digital watermarkdetection status.

FIGS. 3A-4B are figures showing embodiment outline B. FIG. 3A shows aflowchart of a digital watermark embedding method of the embodimentoutline B. This method is a digital watermark embedding method in adigital watermark embedding apparatus for embedding digital watermarkinto moving images. In the method, moving image data including a frameimage group, watermark information, and watermark pattern switchinginformation that is period information for designating phase change ofwatermark pattern are supplied first. Then, the digital watermarkembedding apparatus sequentially obtains each frame image of the movingimage data and its frame display time (step B1). Then, the digitalwatermark embedding apparatus obtains a watermark pattern using thewatermark information, the frame display time and the watermark patternswitching information (step B2) and superimposes the watermark patteronto the frame image (step B3). Then, the watermark embedding apparatuscombines the watermark embedded frame images by sequentially repeatingthe step B1-step B3 so as to generate watermark embedded moving imagedata (step B4).

FIG. 3B shows a flowchart of a digital watermark detection method of theembodiment outline B. This method is a digital watermark detectionmethod by a digital watermark detection apparatus for detecting digitalwatermark from the moving images. In the method, a frame image issequentially obtained, first (step B11). Next, the digital watermarkdetection apparatus generates a difference image (A) between a currentlyobtained frame image and a previously obtained frame image that is readfrom image storing means, and stores the currently obtained frame imageinto the image storing means (step B12). The digital watermark detectionapparatus generates a phase pattern based on the currently obtaineddifference image (A), measures a phase difference between the currentlyobtained phase pattern and a previously obtained phase pattern that isread from the phase pattern storing means so as to generate a differenceimage (B) based on the phase difference and stores the currentlyobtained phase pattern into the phase pattern storing means (step B13).Then, the digital watermark detection apparatus performs digitalwatermark detection from the difference image (B) to output digitalwatermark detection status (step B14). Then, in a case when continuingthe digital watermark detection process, the digital watermark detectionapparatus obtains a new frame image again and repeats theabove-mentioned processes (step B15).

FIG. 4A shows a block diagram of a digital watermark embedding apparatusof the embodiment outline B. This digital watermark embedding apparatusincludes frame image obtaining means B110 configured to sequentiallyobtain each frame image of moving image data including input frameimages and display time of the frame image, pattern generation meansB120 configured to generate a watermark pattern using the suppliedwatermark information, the frame image display time and watermarkpattern switching information that is period information for designatingphase change of watermark pattern, watermark pattern superimposing meansB130 configured to superimpose the watermark patter onto the frameimage, and moving image data reconstruction means B140 configured tocombine the watermark embedded frame images so as to generate and outputwatermark embedded moving image data.

FIG. 4B shows a block diagram of a digital watermark detection apparatusof the embodiment outline B. This digital watermark detection apparatusincludes moving image input means B210 configured to sequentially obtaina frame image, difference image generation means B230 configured togenerate a difference image (A) between a currently obtained frame imageand a previously obtained frame image that is read from image storingmeans, and stores the currently obtained frame image into the imagestoring means B250, phase difference calculation means B360 configuredto generate a phase pattern based on the currently obtained differenceimage (A), measure a phase difference between the currently obtainedphase pattern and a previously obtained phase pattern that is read fromthe phase pattern storing means B370 so as to generate a differenceimage (B) based on the phase difference and stores the currentlyobtained phase pattern into the phase pattern storing means, and digitalwatermark detection means B240 configured to perform digital watermarkdetection from the difference image (B) to output digital watermarkdetection status.

FIGS. 5A-6B shows an embodiment outline C. FIG. 5A shows a flowchart ofa digital watermark embedding method of the embodiment outline C. Thismethod is a digital watermark embedding method in a digital watermarkembedding apparatus for embedding digital watermark into moving images.In the method, moving image data including a frame image, watermarkinformation, and watermark pattern switching information are supplied tothe digital watermark apparatus first (step C1). Then, the digitalwatermark embedding apparatus sequentially obtains each frame image ofthe moving image data and its frame display time (step C2). Then, thedigital watermark embedding apparatus generates a watermark patternusing the watermark information, the frame display time and thewatermark pattern switching information (step C3). The digital watermarkembedding apparatus changes scale of the watermark pattern into a sizeequal to or smaller than the frame image and superimposes the watermarkpatter onto the inside of the frame image (step C4), and combines thewatermark embedded frame images by sequentially repeating the stepC2-step C4 so as to generate watermark embedded moving image data (stepC5).

FIG. 5B shows a flowchart of a digital watermark detection method of anembodiment outline C. This method is a digital watermark detectionmethod in a digital watermark detection apparatus for detecting digitalwatermark from the moving images. In the method, a frame image issequentially obtained, first (step C6). Next, the digital watermarkdetection apparatus generates a difference image between a currentlyobtained frame image and a previously obtained frame image (step C7),and extracts detection subject region from the difference image togenerate a detection subject region image (step C8). Then, the digitalwatermark detection apparatus performs digital watermark detection forthe detection subject region image to output digital watermark detectionstatus (step C9), and in a case when continuing the digital watermarkdetection process, the digital watermark detection apparatus obtains anew frame image again and repeats the above-mentioned processes (stepC10).

FIG. 6A is a block diagram of the digital watermark embedding apparatusof the embodiment outline C. The digital watermark embedding apparatusincludes frame image obtaining means C110 configured to sequentiallyobtain each frame image of the moving image data and its frame displaytime, watermark pattern generation means C120 configured to generate awatermark pattern using the input watermark information, the watermarkpattern switching information and the frame display time, watermarkpattern superimposing means C130 configured to change scale of thewatermark pattern into a size equal to or smaller than the frame imageand superimposes the watermark patter onto the inside of the frameimage, and moving image data reconstruction means C140 configured tocombine the watermark embedded frame images by sequentially repeatingthe above-mentioned processes so as to generate watermark embeddedmoving image data.

FIG. 6B shows a block diagram of the digital watermark detectionapparatus of the embodiment outline C. The digital watermark detectionapparatus includes moving image input means C210 configured tosequentially obtain a frame image, difference image generation meansC230 configured to generate a difference image between a currentlyobtained frame image and a previously obtained frame image, and digitalwatermark, detection subject region extraction means C220 configured toextract a detection subject region from the difference image to generatea detection subject region image, and detection means C240 configured toperform digital watermark detection from the detection subject regionimage to output digital watermark detection status.

First Embodiment

<Digital Watermark Embedding Apparatus>

FIG. 7 shows a configuration of the digital watermark embeddingapparatus of the first embodiment of the present invention.

The digital watermark embedding apparatus 100 shown in the figureincludes a frame image obtaining unit 110, a watermark patterngeneration unit 120, a watermark pattern superimposing unit 130 and amoving image data reconstruction unit 140.

The frame image obtaining unit 110 receives original moving image data,and the watermark pattern generation unit 120 receives watermarkinformation and watermark pattern switching information. In addition,the moving image data reconstruction unit 140 outputs watermark embeddedmoving image data.

In the following, operation of the digital watermark embedding apparatus100 is described.

FIG. 8 shows a flowchart of the operation of the digital watermarkembedding apparatus in the first embodiment of the present invention.

Step 110) As shown in FIG. 9, the frame image obtaining unit 110sequentially obtains a frame image and the frame display time from theoriginal moving image data one by one. The frame display time is, forexample, may be one indicating absolute time, from the head of themoving images, that is determined from time code and frame rate forreproducing, or may be one by which relative time interval betweenframes for reproducing can be measured such as sequential serial numberassigned to each frame. When the input moving image data is coded datasuch as MPEG data, the frame image obtaining unit 110 obtains the frameimage after performing decoding.

Step 120) The watermark pattern generation unit 120 receives thewatermark information and the watermark switching information so as togenerate a watermark pattern using the watermark information, framedisplay time and watermark pattern switching information.

Step 130) The watermark pattern superimposing unit 130 superimposes thewatermark patter generated by the watermark pattern generation unit 120onto the frame image to generate a watermark embedded frame image.

Step 140) Finally, the moving image data reconstruction unit 140reconstructs a series of watermark embedded frame images that aresequentially generated as moving image data so as to output it aswatermark embedded moving image data. At this time, encoding such asMPEG encoding may be performed as necessary.

Next, the watermark pattern generation unit 120 is described in detail.

FIG. 10 shows a configuration of the watermark pattern generation unitin the first embodiment of the present invention.

The watermark pattern generation unit 120 includes a basic watermarkpattern generation unit 120 and a watermark pattern switching unit 122.

FIG. 11 is a flowchart of operation of the watermark pattern generationunit in the first embodiment of the present invention.

Step 121) When receiving the watermark information, the basic watermarkpattern generation unit 121 converts the watermark information into abasic watermark pattern that is a two-dimensional pattern.

As shown in FIG. 12, as the method for the conversion, a method that isusually used for digital watermark technique for still images, such as atechnique described in “Nakamura, Katayama, Yamamuro, Sonehara: Highspeed digital watermarking scheme from analog image usingcamera-equipped mobile-phone, IEICE, (D-II), Vo. J87-D-II, No. 12, pp.2145-2155, December 2004 (to be referred to as document 1 hereinafter),for example, can be used. The methods include a method for associatingsizes of pixel values with series values obtained by directly performingspread-spectrum modulation on bit values of watermark information usingspreading series (FIG. 12: modulation method (A-1)), a method forswitching waveform patterns according to the series values (FIG. 12:modulation method (A-2))), a method for associating the value of a termof the series with two blocks to represent the value using presence orabsence of change of pixel values (FIG. 12: modulation method (A-3)),and a method for representing the value using presence or absence ofchange of phase of waveform pattern (FIG. 12: modulation method (A-4)).By the way, when using modulation such as (FIG. 12: A-3) and (FIG. 12:A-4), modulation can be performed while shifting the position of twoblocks corresponding to a term of series one-block by one-block suchthat blocks are overlapped.

Or, as shown in the example of FIG. 13, a method can be considered fordividing watermark information into symbols each having a certain length(one bit per symbol in the example of FIG. 13), associating eachpossible value of the symbol with different independent spreadingseries, combining the spreading series or multiplexing the spreadingseries by superimposition (not shown in the figure) to obtain seriesvalues, and converting the series values into a two-dimensional patternin the same way as the case of FIG. 12.

By the way, it is desirable to use large sized block or use a lowfrequency pattern as a waveform pattern for the basic watermark patternin consideration of a case where resolution for camera capturing becomeslow. In addition in the examples shown in FIGS. 12 and 13, although thepixel pattern is directly obtained from a result obtained by performingspread spectrum modulation on the watermark information, the imagepattern may be obtained by performing similar process in a frequencyspace and by using inverse orthogonal transform as shown in document 2“T. Nakamura, H. Ogawa, A. Tomioka, Y. Takashima, “Improved digitalwatermark robustness against translation and/or cropping of an imagearea”, IEICE Trans. Fundamentals, vol. E83-A, No. 1, pp. 68-76, January2000” (to be referred to as document 2). Anyway, any method can be usedas long as it is a watermark scheme of a type in which watermark issuperimposed on the image in an adding manner.

In the following, for the sake of explanation, it is assumed that eachpixel value of the basic watermark pattern has a value of plus or minus,and that an average of the pixel values is 0. It is obviously easy torealize such values by shifting an average value of an arbitrary basicwatermark pattern.

Step 122) Next, the watermark pattern switching unit 122 in thewatermark patter generation unit 120 determines necessity of phasereversal of the basic watermark pattern based on relationship betweenthe frame display time and the pattern switching information.

Step 123) When the phase reversal is necessary, the watermark patterngeneration unit 120 reverses the phase of the basic watermark pattern tooutput it as a watermark pattern.

Step 124) When the phase reversal is unnecessary, the watermark patterngeneration unit 120 outputs the basic watermark pattern as it is as thewatermark pattern.

FIGS. 14A-14C shows examples of watermark pattern switching informationin the first embodiment of the present invention. The watermark patternswitching information is information showing how the phase of the basicwatermark pattern is changed with respect to a time axis for reproducingthe moving images, and may be one that controls the basic watermarkpattern to reverse at constant time intervals as shown in FIG. 14A, maybe one that controls the basic watermark pattern to reverse randomly ina unit of time as shown in FIG. 14B, or may be one that controls thebasic watermark pattern to reverse at complicated timing. By using acomplicated pattern such as random switching, analyzing the patternbecomes difficult. Thus, when the present invention is used for securitysuch as copyright protection, security against attack to digitalwatermark improves. By the way, although the examples shown in FIGS.14A-14C show only cases where two values of phase 0 and π are taken likea rectangle, a pattern in which the phase changes smoothly such as asine curve can also be used, for example.

FIG. 15 is a figure for explaining processes of the watermark patternswitching unit in the first embodiment of the present invention. Thefigure shows a case where the before-mentioned modulation method (A-1)is used. The watermark pattern switching unit 122 receives the watermarkpattern switching information, the frame display time and the basicwatermark pattern, and it is assumed that the watermark patternswitching information is information for instructing reversal for each1/10 second. When it is assumed that the original moving images arereproduced at 30 frames per second, the phase of the image is reversedevery three frames. If it is assumed that, when the frame display time tobtained by the frame image obtaining unit 110 is placed on a time axisof the watermark pattern switching information, the phase represented bythe watermark pattern switching information is π, watermark pattern atthe frame display time is determined to be one obtained by reversing thephase of the basic watermark pattern.

By the way, the basic watermark pattern has plus and minus pixel values,and the average value is 0. Therefore, phase reverse can be obtained bymultiplying each pixel value by −1. If the phase is 0 when a framedisplay time t′ is placed on the time axis of the watermark patternswitching information, the basic watermark pattern with phase changeamount 0, that is, without change is output as the watermark pattern.

By the way, as shown in FIG. 16, since phase modulation in the watermarkpattern switching unit 122 is performed for amplitude of pixel value,the pixel value is changed in which direction of waveform is not changedin the case of the basic watermark pattern in which a waveform patternis used for each block like the modulation method (A-2). In addition, itis desirable that phase change of the watermark pattern represented bythe watermark pattern switching information does not include bias to thepositive phase or to the reverse phase within a range of a certain timeinterval (1 second, for example) in order to suppress image qualitydegradation of the watermark embedded moving images. That is, it isdesirable that integral of a time section in the lateral axis of thegraph in which the phase bias is 0 becomes 0.

As to watermark pattern switching for reversing or non-reversing, it isdesirable that sum of time of reversal and sum of time of non-reversalwithin a time section become almost the same.

Next, the watermark pattern superimposing unit 130 is described indetail.

FIG. 17 is a figure for explaining processes of the watermark patternsuperimposing unit in the first embodiment of the present invention.

The watermark pattern superimposing unit 130 receives a frame image anda watermark pattern generated based on the frame display time. First,the watermark pattern superimposing unit 130 enlarges the watermarkpattern to a size of the frame image, and adds the enlarged watermarkpattern to the frame image to obtain a watermark embedded frame image.It is possible to adjust tradeoff balance between tolerance of thedigital watermark and image quality deterioration by increasing ordecreasing amplitude of the watermark pattern as necessary beforeadding. In addition, when adding the watermark pattern onto the frameimage, there may be various methods as to which component it is addedto. For example, the pattern is added to luminance values of the frameimage, or added to Blue component in the RGB color coordinate system.Or, the pattern is added to the Yellow component in the CMYK colorcoordinate system, is added to the Hue component of the HSV colorcoordinate system, or added to the Cb component in the YCbCr colorcoordinate system. In addition, there may be a case where the watermarkpattern uses a plurality of components for one pixel in which a pixel inthe watermark pattern has two values not one value, for example. In sucha case, there may be a case where a plurality of components are changedby adding pixel values of each plane of the watermark pattern onto theRed component and the Green component of the RGB color coordinatesystem, for example.

Finally, the moving image data reconstruction unit 140 is described indetail.

The moving image data reconstruction unit 140 reconstructs the watermarkembedded frame images that are sequentially generated according to theabove-mentioned processes and outputs the reconstructed data aswatermark embedded moving image data. When performing reconstruction,encoding such as MPEG encoding may be performed.

The watermark embedding apparatus 100 in the present embodiment has beendescribed so far.

Here, a using scene of the present embodiment is described.

FIG. 18 is a figure for explaining the using scene of the firstembodiment of the present invention.

The watermark embedded moving image data described above is beingreproduced on a TV display apparatus by TV broadcasting and the like(1). When a viewer uses an information service related to the watchingprogram, image-taking is started by directing a camera such as acamera-equipped mobile-phone to a display screen of the TV (2). Bytaking the images, video frames are captured sequentially so as to beentered into the mobile phone in real time, so that detection of digitalwatermark from the frame group is performed (3). Based on detecteddigital watermark information, a related information service such asobtaining information resources on the network can be utilized, forexample (4).

<Digital Watermark Detection Apparatus>

Next, a digital watermark detection apparatus is described.

FIG. 19 shows a configuration of the digital watermark detectionapparatus in the first embodiment of the present invention.

The digital watermark detection apparatus shown in the figure includesan moving image input unit 210, a detection subject region extractionunit 220, a difference image generation unit 230, a digital watermarkdetection unit 240, and a detection subject region image buffer 250. Themoving image input unit 210 receives analog moving images displayed on aTV and the like, or receives MPEG-encoded digital moving images, and thedigital watermark detection unit 240 outputs a detection result.

FIG. 20 is a flowchart of operation of the digital watermark detectionapparatus in the first embodiment of the present invention.

Step 201) When the analog or digital moving images are input by themoving image input unit 210, a frame image is obtained sequentially.When inputting the analog moving images, camera, scanner, or analogvideo signals are input so that the frame image is obtained. When thedigital moving images are input, the frame image is obtained afterperforming decoding processes.

Step 202) Next, the detection subject region extraction unit 220extracts a detection subject, in each frame image that is inputsequentially, that is a subject from which watermark is detected, so asto obtain the detection subject region image.

Step 203) Next, the difference image generation unit 230 generates adifference image between the currently obtained detection subject regionimage and a previously obtained detection subject region image stored inthe detection subject region image buffer 250.

Step 204) The difference image generation unit 230 buffers the currentdetection subject region image into the detection subject region imagebuffer 250 in preparation for next detection trial.

Step 205) The digital watermark detection unit 240 tries to detectdigital watermark from the difference image and outputs a detectionresult. When the digital watermark detection does not succeed, themoving image input unit 210 obtains a next frame image so as to repeatthe above-mentioned processes sequentially.

Next, processes of the detection subject region extraction unit 220 aredescribed in detail.

FIG. 21 is a figure for explaining processes of the detection subjectregion extraction unit in the first embodiment of the present invention.

The detection subject region extraction unit 220 receives the frameimage. FIG. 21 shows an example of camera input. In the case of thecamera input, the detection subject region in the frame image suffersgeometric deformation in various forms so that it changes due to planeprojection conversion caused by image-taking angle or hand jiggling.Therefore, the detection subject region extraction unit 220 detects adisplay region of a TV and the like for each captured frame image usingan edge detection/corner detection method such as “Katayama, Nakamura,Yamamuro, Sonehara: “i appli high speed corner detection algorithm forreading digital watermark”, IEICE (D-II), Vol. J88-D-II, No. 6, pp.1035-1046, June, 2005 (referred to as document 3, in the following).Plane projection distortion of the detected display area is correctedand the image size is normalized to a constant size so that thedetection subject region image is output for generating the differenceimage next. By the way, in the case of analog video signal input such asNTSC or digital moving image input such as MPEG, processes for removingdistortion in the frame or cutting out a part of the frame image areunnecessary, and the frame image is output as it is as the detectionsubject region image.

Next, the difference image generation unit 230 is described in detail.

FIG. 22 is a figure for explaining processes of the difference imagegeneration unit in the first embodiment of the present invention.

The difference image generation unit 230 receives the detection subjectregion image. The difference image generation unit 230 generates adifference image between the currently received detection subject regionimage and a previously obtained detection subject region image stored inthe detection subject region image buffer 250. Most simply, it can beconsidered that a next previous detection subject image is buffered asthe subtraction subject. Also, a plurality of detection subject regionimages may be buffered to select a subtraction subject as necessary togenerate the difference image. In addition, in preparation for nextdetection trial, the detection subject region image buffer 250 buffersthe current detection subject region image. At this time, an olddetection subject region image in the detection subject region imagebuffer 250 may be discarded.

By the way, at the timing when the first detection subject region imageis input, the detection subject region image buffer 250 is empty. Inthis case, the first detection subject region image may be output as thedifference image to try to perform digital watermark detection, ordigital watermark detection may be skipped to perform next framecapturing process.

Next, the digital watermark detection unit 240 is described in detail.

The digital watermark detection unit 240 receives the difference imageand tries to detect digital watermark from the difference image.

FIG. 23 is a figure for explaining an example of processes of thedigital watermark detection unit in the first embodiment of the presentinvention. The figure shows a demodulation method when performingdetection corresponding to the modulation method (B-1) used for digitalwatermark embedding.

First, a component channel used for embedding for the difference imageis extracted. If it is luminance, a luminance component is extracted.Then, the difference image is divided into blocks a number of which isthe same as the number of blocks of the watermark pattern whenembedding.

Next, sum of luminance values in each block is obtained, and sums arearranged in an order of blocks to obtain a detection subject series thatis a one-dimensional series. In the case of modulation information(B-2), energy of the waveform pattern in each block is obtained, and anindicator indicating a dominant pattern in a plurality of waveformpatterns is calculated for each block, so that the indicators arearranged in a line (document 1).

Corresponding to that one bit symbol is modulated for each section ofthe series when embedding, sections for each symbol position of thedetection subject series are extracted so as to perform correlationcalculation for a spreading series representing a symbol value “0” and aspreading series representing “1” for each section. In each symbolposition, a symbol value by which (absolute value of) the correlationvalue is maximum becomes detected watermark information for each symbolposition. In the example shown in FIG. 23, the first symbol: “0” and thesecond symbol: “1” are detected watermark information. In addition, itis determined that digital watermark detection succeeds when (absolutevalue of) the correlation value obtained at this time is sufficientlylarge, that is, when the correlation value is equal to or greater than apredetermined threshold, for example. By the way, as to detectionmethods for the modulation methods (A-1)-(A-4), bit values can bedetermined using plus and minus of the correlation value as shown in thedocument 1.

When the digital watermark detection unit 240 successfully detects thedigital watermark, the detected watermark information may be output soas to complete the detection process and change the state to providing arelated information service. Alternatively, when using an augmentedreality application in which taken images and a CG (computer graphics)are synthesized as described in “J. Rekimoto, Y. Ayatsuka, “CyberCode:Designing Augmented Reality Environments with Visual Tags”, DesigningAugmented Reality Environments (DARE 2000), 2000” (to be referred to asdocument 4 hereinafter), digital watermark detection may be continued.When the digital watermark detection is not successfully performed, theprocedure retunes to the frame capturing process of the moving imageinput unit 210 so as to repeat detection trial. Or, when input of movingimages completes, the detection process is terminated.

Effects of the Present Embodiment

According to the present embodiment, when embedding digital watermarkinto the moving images, embedding is performed while switching thewatermark pattern for each frame. Thus, compared with an embeddingmethod in which a fixed watermark pattern is embedded in each frameimage by simply repeating applying a still image algorithm on the frameimage, the fixed pattern can be avoided so that image qualitydeterioration can be kept low. Especially, when the switching speed ishigh, image quality deterioration becomes less perceptible in view ofthe knowledge on time frequency characteristics that visual sensitivitydecreases when time frequency is high in visual characteristics of ahuman (“Okewatari, Image engineering handbook, Asakurashoten,ISBN4-254-20033-1, pp. 10-56, 1986”, (to be referred as document 5,hereinafter).

In addition, in the digital watermark detection process, timesubtraction is calculated for the captured frame images to detectdigital watermark from the difference image. As shown in FIG. 24, it isgenerally known that inter-frame correlation is high in a time directionin moving images. Thus, by calculating subtraction, almost all originalimage components except for a part including intense movements can becanceled. At the same time, in a case of subtraction between two framesin which watermark patterns are reversed, a watermark pattern whoseamplitude is twice as that of a watermark pattern used for embedding canbe obtained. Accordingly, the original image signal that is a largenoise component for a watermark signal can be suppressed and further theamplitude of the watermark signal can be doubled, which were challengescommon to digital watermark schemes of a type in which a watermarkpattern is additively superimposed onto an image. Therefore, an S/Nratio of the watermark signal can be largely improved when performingdetection, so that tolerance of digital watermark improves.

In addition, the detection subject region in the captured frame image isextracted, distortion is corrected and normalization is performed beforegenerating the difference image. Thus, various distortion factors in thedetection subject region in the captured frame can be canceled, whereinthe factors are such as plane projection conversion distortion due to animage-taking angle when performing camera input, and paralleltranslation due to camera movement and viewpoint movement and the like.Thus, digital watermark detection from camera input can be realized.Further, by performing the correction for each capture frame, differentdistortion factors for each taken frame due to movement of the cameracaused when the camera is held by hands can be canceled. Thus, theoriginal image signal can be reliably suppressed by generating thedifference image, so that watermark detection can be successfullyperformed without any problem even in a case when taking images byholding a camera-equipped mobile-phone by hand.

Second Embodiment

In this embodiment, a digital watermark detection apparatus isdescribed.

This embodiment is the same as the before-mentioned first embodimentexcept for parts described below.

FIG. 25 shows a configuration of the digital watermark detectionapparatus in the second embodiment of the present invention. The digitalwatermark detection apparatus 200B shown in the figure is configuredsuch that a detection necessity determination unit 206 is added to theconfiguration shown in FIG. 15 in the before-mentioned first embodiment.

FIG. 26 shows a flowchart of operation of the digital watermarkdetection apparatus in the second embodiment of the present invention.

Step 301) The moving image input unit 210 of the digital watermarkdetection apparatus 200B inputs analog or digital moving images andobtains a frame image sequentially, and at the same time, obtains theframe display time. The frame display time can be obtained based on timecode or frame rate or the like in a case of MPEG-encoded moving images,for example. In a case of a camera and the like, it can be consideredthat synchronization with a display system cannot be taken. In such acase, a timer may be provided in the moving image input unit 210 forobtaining time when capturing or obtaining delay time from next previouscapturing so that the frame display time of the captured frame can beobtained.

Step 302) Next, the detection subject region extraction unit 220extracts a detection subject region, in each of the sequentially inputframe images, that is a subject from which watermark is detected toobtain a detection subject region.

Step 303) Next, the detection necessity determination unit 260 comparesa frame display time corresponding to the currently obtained detectionsubject region image with a frame display time corresponding to adetection subject region image obtained before and stored in thedetection subject region image buffer while referring to watermarkpattern switching information the same as that used when embedding. Whenthe watermark patterns in the two detection subject region images are inphase, it is judged that detection is unnecessary so that the proceduregoes to step 304. When it is judged that they are not in phase, theprocedure goes to step 305.

Step 304) When detection is unnecessary, the procedure is returned tothe capturing process by the moving image input unit 210 after thecurrent detection subject region image and corresponding frame displaytime are buffered in the detection subject region image buffer 250 (goto step 301).

Step 305) The current detection subject region image is sent to thedifference image generation unit 230. The difference image generationunit 230 generates a difference image between the currently obtaineddetection subject region image and the previously obtained detectionsubject region image stored in the detection subject region image buffer250.

Step 306) The difference image generation unit 230 buffers the currentdetection subject region image and the frame display time in thedetection subject region image buffer 250 in preparation for a nextdetection trial. At this time, old detection subject region images andframe display times in the detection subject region image buffer arediscarded for example.

Step 307) Next, the digital watermark detection unit 240 tries to detectdigital watermark from the difference image and outputs a detectionresult.

Step 308) It is determined whether digital watermark detection succeeds.When it does not succeed, the moving image input unit 210 obtains a nextframe image to repeat the above-mentioned process sequentially (go tostep 301). When the digital watermark detection succeeds, the process isterminated.

The detection necessity determination unit 260 receives the framedisplay time, the detection subject region image and the watermarkpattern switching information. The detection necessity determinationunit 260 compares a frame display time corresponding to the currentlyobtained detection subject region image with a frame display timecorresponding to a previously obtained detection subject region imagestored in the detection subject region image buffer 250 while referringto watermark pattern switching information the same as that used whenembedding. The detection necessity determination unit 260 determinesdetection necessity by determining whether watermark patterns in the twodetection subject region images are in phase. When detection isunnecessary, the procedure returns to capturing process by the movingimage input unit 210. When it is determined that detection is necessary,the detection necessity determination unit 260 sends the detectionsubject region images to the difference image generation unit 230.

FIG. 27 is a figure for explaining processes of the detection necessitydetermination unit in the second embodiment of the present invention.

As an example, it is assumed that the watermark pattern switchinginformation indicates “switch every 1/10 second”, and that the movingimages are reproduced 30 frames per second. In addition, the framedisplay time of the previous frame stored in the detection subjectregion image buffer 250 is indicated by t. When the currently obtainedframe display time is t0 or t1, each of t0−t= 3/30 second and t1−t= 5/30second indicates that the phase of the watermark pattern is reversed inthe every 1/10 second reversal so that it can be ascertained thatpossibility in which digital watermark can be detected from thedifference image is high. Thus, it is judged that “detection isnecessary”. When the currently obtained frame display time is t2, t2−t=6/30 second indicates that the phase of the watermark pattern is inphase with the previous display time in the every 1/10 second reversalso that it can be ascertained that possibility in which digitalwatermark can be detected from the difference image is low. Thus, it isjudged that “detection is unnecessary”.

Like this example, it is not necessary that the frame display time is anabsolute time from the head of the moving images, and it is onlynecessary that a delay time between capturing times of the reproducedmoving images.

Effects of the Present Embodiment

According to this embodiment, in addition to the effects of the firstembodiment, useless watermark detection process can be omitted byperforming the detection necessity determination so that efficientdetection process can be performed. As shown in FIGS. 28A and B, sincethe difference image is generated to perform detection trial only whenthe phases of the watermark patterns are reversed by evaluating thetiming of capturing, the in-phase case in which detection is difficultcan be avoided so that useless detection trial can be omitted. Thereduced process amount can be used for improving detection performanceby increasing the frame rate of capturing and for advancing the userinterface so that effects such as improving convenience of the detectionapparatus can be obtained.

Third Embodiment

In the present embodiment, a digital watermark detection apparatus isdescribed.

This embodiment is the same as the first embodiment except for partsdescribed as follows.

FIG. 29 shows a configuration of the digital watermark detectionapparatus in the third embodiment of the present invention.

The digital watermark detection apparatus 200C shown in the figure isconfigured such that a capture timing control unit 270 is added to theconfiguration shown in FIG. 19 in the first embodiment.

The capture timing control unit 270 receives watermark pattern switchinginformation and sends a capture timing signal to the moving image inputunit 210. In addition, the capture timing control unit 270 sends adifference timing signal to the difference image generation unit 230.

FIG. 30 shows a flowchart of operation of a digital watermark detectionapparatus in the third embodiment of the present invention.

Step 401) The moving image input unit 210 inputs analog or digitalmoving images at capturing time intervals specified by a capture timingsignal supplied from the capture timing control unit 270 so as tosequentially obtain a frame image.

Step 402) The detection subject region extraction unit 220 extracts adetection subject region, in each of the sequentially input frameimages, that is an object from which watermark is detected to obtain adetection subject region image.

Step 403) Next, the difference image generation unit 230 generates adifference image between the currently obtained detection subject regionimage and the temporally previous detection subject region image that isindicated by the difference timing signal and that is stored in thedetection subject region image buffer 250.

Step 404) In addition, the difference image generation unit 230 buffersthe current detection subject region image in the detection subjectregion image buffer 250 in preparation for a next detection trial.

Step 405) Next, the digital watermark detection unit 240 tries to detectdigital watermark from the difference image and outputs a detectionresult.

Step 406) When digital watermark detection does not succeed, the movingimage input unit 210 obtains a next frame image to repeat theabove-mentioned process sequentially (go to step 401). When the digitalwatermark detection succeeds, the process is terminated.

FIG. 31 is a figure for explaining processes of the capture timingcontrol unit in the third embodiment of the present invention. As anexample, it is assumed that the watermark pattern switching informationis “switch every 1/10 second”, and that the moving images are reproduced30 frames per second. In this case, the capture timing signal is sentevery 1/10 second, and the moving image input unit 210 that receives itperforms capturing at intervals of 1/10 second. Alternatively,information indicating “capture at intervals of 1/10 second” is sent asthe capture timing signal first, so that capturing operation may beperformed at intervals of 1/10 second while adjusting timing in themoving image input unit 210.

In addition, as the difference timing signal, an instruction indicating“obtain difference between detection subject region images (odd numbertimes of) 1/10 second apart” is sent to the difference image generationunit 230. When using camera input and the like, there may be a case inwhich actually captured frames are different when only time interval forcapturing is specified (deference between a capture example A and acapture example B in FIG. 31). However, in either of the capturingexamples, frames in which watermark patterns having phases reversed witheach other are embedded can be alternately captured according tocapturing every 1/10 second. Accordingly, possibility of detectingdigital watermark can be made high from a difference between any twoconsecutive frames. That is, it does not depend on start timing forcapturing.

In addition, as shown in FIG. 32, when it is instructed to “performcapturing at intervals of 1/30 second and obtain difference betweendetection subject region images (odd number times of) 1/10 second apart”and when the detection subject region image buffer 250 buffers previousthree detection subject region images obtained from frames that arecaptured every 1/30 second, possibility of detection can be made high inevery watermark detection trial by sequentially obtaining differencebetween a new detection subject region image input every 1/30 second anda detection subject region image 1/10 seconds before, that is, threeimages before.

By the way, when input data are digital data such as MPEG or when theinput data are analog video signals, that is, when the input framedisplay time can be accurately obtained, since it is obvious thatsynchronization can be maintained, the possibility of detection can bemade high in each detection trial by performing processes the same asthose described above.

Effects of the Present Embodiment

According to the present embodiment, in addition to the effects of thefirst embodiment, capturing and difference image generation areperformed at timing synchronized with the watermark pattern switchingtiming as shown in FIGS. 33A and B so that detection can be performed byobtaining a difference between images in which the phases of thewatermark patterns are always reversed with each other. Thus, detectioncan be performed from difference images in which the watermark patternis enhanced in all detection trials so that detection possibility can beincreased to be high. Accordingly, since useless trial of low detectionpossibility can be eliminated, time required for detecting a digitalwatermark can be reduced, and detection can be succeeded quickly afterstarting capturing. In addition, in a case where detection is continuedalthough watermark detection is succeeded like a case in an augmentedreality application, since a detection succeed status continues withoutinterruption, effects in which user interface is improved andconvenience is improved can be also obtained.

Fourth Embodiment

The present embodiment describes a method for detecting correctwatermark information even when using a modulation method of a type inwhich bits of detected watermark information are reversed due toreversal of the phase of the watermark pattern.

This embodiment is the same as the embodiments in the first-thirdembodiments except for parts described below.

FIGS. 34A and 34B shows process contents of the basic watermark patterngeneration unit in the fourth embodiment of the present invention. Thefigures show process contents of the basic watermark pattern generationunit 121 in the digital watermark embedding apparatus 100 shown in FIG.10 in the first embodiment.

The basic watermark pattern generation unit 121 in each of the first tothird embodiments modulates only the input watermark information to abasic watermark pattern. In this embodiment, the basic watermark patternis generated by adding one bit that is a flag bit for determining bitreversal of watermark information to the input watermark information.Alternatively, assuming that bit length of watermark information is onebit less than that of the watermark information of the first to thirdembodiments, the flag bit for determining the bit reversal is added sothat a basic watermark pattern may be generated based on informationwhose length is the same as the watermark information of the first tothird embodiments. In the example shown in FIG. 34, a bit value “0” asthe flag bit for determining bit reversal is added to indicate apositive phase, that is, indicate that there is no bit reversal.

FIG. 35 is a flowchart of operation of the digital watermark detectionunit in the fourth embodiment of the present invention.

Step 501) The digital watermark detection unit 240 receives thedifference image, and tries to detect digital watermark in the same wayas the first to third embodiments.

Step 502) When the digital watermark is detected, the process goes tostep 504. When it is not detected, the process goes to step 503.

Step 503) When it is not detected, operation is returned to next framecapturing.

Step 504) When the digital watermark is detected, a bit reversaldetermination flag in the detected information is checked and it isevaluated whether the value represents the positive phase. When itrepresents the positive phase, the process goes to step 506. When itrepresents a reverse phase, the process goes to step 505.

Step 505) When the bit reversal determination flag indicates reversedphase, bits of information from which the bit reversal determinationflag is removed are reversed, and the reversed watermark information isoutput as detected watermark information as a detection result.

Step 506) When the value indicates positive phase, information obtainedby removing the bit reversal determination flag from the detectedinformation is regarded to be watermark information, and the detectionresult is output.

By the way, like the third embodiment, it is obvious that a phase of awatermark pattern in a current difference image can be also determinedby using watermark pattern switching information used when embedding andthe frame display time used when capturing.

Effects of the Present Embodiment

Effects of the present embodiment are described using FIGS. 36-38.

When the modulation method used for digital watermark embedding is themodulation method (A-1) shown in FIG. 12 described in the firstembodiment, detection subject series are obtained from pixel values ofthe difference image and correlation between the detection object seriesand spreading series is calculated so as to detect a bit value “0” or“1” based on plus/minus of the correlation value when performingdetection. Demodulation in the case of the modulation method (A-1) isdescribed with reference to FIG. 36. A basic watermark pattern obtainedby performing modulation is embedded in frame images of moving images byreversing the phase as necessary. Since various noises are added to thewatermark pattern that is embedded while the watermark embedded movingimage is encoded or analog display is captured or the like, signals ofthe detection subject series obtained from the difference image arechanged compared with watermark information that is modulated whenembedding. However, by calculating correlation with the spreading seriesused when embedding, a correlation value of a section corresponding to abit “1” should become a positive large value, and a correlation value ofa section corresponding to a bit “0” should become a minus large value.Accordingly, embedded bit information can be demodulated.

However, as shown in FIG. 37, the phase of the basic watermark patternis switched to positive phase/reverse phase in a time direction by thewatermark pattern switching unit 122. Therefore, for example,considering a difference image obtained from frames of (i) and (ro)shown in FIG. 37, since a reversed phase of the basic watermark patternis embedded in (i) and an in-phase with the basic watermark pattern isembedded in (ro), a difference image obtained by (ro)−(i) becomes anin-phase, that is, a positive phase with the basic watermark pattern, sothat watermark information can be detected correctly. However, dependingon timing of capturing, there may be a case of obtaining a frame imagefrom frames of (ha) and (ni). In this case, since (ha) is in phase withthe basic watermark pattern and (ni) has the reversed phase of the basicwatermark pattern, the difference image becomes reversed phase of thebasic watermark pattern. At this time, if watermark detection isperformed, since plus/minus of the value of each item of the detectionsubject series described in FIG. 36 is reversed, a sign of thecorrelation value is reversed. That is, there may occur a problem inthat, even though watermark information “01” is embedded, “10” that isbit reversal of the embedded information is correctly detected.

This embodiment is for avoiding this problem, and the bit reversaldetermination flag is added to the embedded watermark information asshown in FIG. 38. Accordingly, it can be determined whether bits ofdetected watermark information are reversed. That is, it can bedetermined whether the phase of the difference image is the same as thatof the basic watermark pattern or is reversed compared with that of thebasic watermark pattern. When the bit reversal flag of the detectioninformation is a value indicating the positive phase, bit values areregarded as watermark information as it is. When it is a valueindicating the reversed phase, bit values of the detection informationare reversed to obtain watermark information. Accordingly, irrespectiveof how two frames are obtained by capturing, correct watermarkinformation can be always detected. This method can be applied to all ofpolarity using type modulation methods.

By the way, as to the modulation methods (A-2)-(A-4) and (B-2)-(B-4)described in the first embodiment, since the detection subject series isnot formed based on (simple total sum) of pixel values of the differenceimage, constant watermark information can always be detectedirrespective of obtaining methods of the difference image. As to amodulation method of a type like the modulation method (B-1) in whichinformation is not represented by plus/minus of a correlation value butinformation can be modulated and demodulated based on spreading serieswith which the information represents a large correlation absolutevalue, a positive correlation value is obtained when the differenceimage is the positive phase, and a minus correlation value is obtainedwhen the difference image is the reversed phase, which are similar to(A-1), but, since detection can be always succeeded without problemirrespective whether the difference image is the positive phase or thereverse phase by determining information, by which an absolute value ofa correlation value is large, to be a detected symbol, such device isnot necessary. Following is a summary.

<Types of Two Dimensional Pattern Modulation and Demodulation>

(1) Pattern amplitude using type: Since a detection subject series isformed based on amplitudes in a pattern in an image, the detectionsubject series is reversed when the difference image is reversed.(Modulation methods (A-1) and (B-1))

(2) Pattern shape using type: Since the detection subject series isformed based on pattern shapes in an image, the detection subject seriesis not reversed even when the difference image is reversed. (Modulationmethods (A-2)-(A-4), and Modulation methods (B-2)-(B-4)).

<Types of Spread Spectrum Modulation and Demodulation>

(a) Polarity using type: A method representing information with apolarity of positive/minus of a correlation value with spreading series.(Modulation methods (A-1)-(A-4)

(b) Polarity non-using type: A method representing information accordingto a series, among a plurality of spreading series, by which the maximumabsolute correlation value is obtained (Modulation methods (B-1)-(B-4))

A modulation method in which bit values are reversed when the differenceimage is reversed is one satisfying (1) and (a), that is, only themodulation method (A-1) in the examples.

Fifth Embodiment

In this embodiment, an example is described in which multiple watermarkembedding of phase synchronization signals is used without using thephase determination flag that is used in the fourth embodiment.

This embodiment is the same as the first to third embodiments except forparts described below.

FIG. 39 is a figure for explaining an example of processes of the basicwatermark pattern generation unit in the fifth embodiment of the presentinvention. This figure indicates process contents of the basic watermarkpattern generation unit 121 of the watermark pattern generation unit 120in the digital watermark embedding apparatus 100. In the basic watermarkpattern generation unit 121 in each of the first to third embodimentsmodulates only input watermark information to the basic watermarkpattern. In this embodiment, not only the input watermark information,but also a phase synchronization signal for specifying a phase of awatermark pattern in a difference image when performing detection isembedded to generate a basic watermark pattern.

In the example shown in FIG. 39, a basic watermark pattern (kou) isobtained using the modulation method (A-1) like the first to thirdembodiments, and further, a basic watermark pattern (otsu) that isobtained by modulating a phase synchronization signal always taking avalue “0” with the modulation method (B-1) is generated as a phasedetermination watermark pattern, and both of them are multiplexed bysuperimposition to obtain a basic watermark pattern that is used forembedding.

A process flow of the digital watermark detection unit 240 in thedigital watermark detection apparatus in the present embodiment isdescribed.

FIG. 40 shows a flowchart of operation of the digital watermarkdetection unit in the fifth embodiment of the present invention.

Step 601) The digital watermark detection unit 240 receives thedifference image, and tries to detect digital watermark from thedifference image like the first to third embodiments.

Step 602) When the digital watermark is not detected, the process goesto step 603. When the digital watermark is detected, the process goes tostep 604.

Step 603) The process returns to next frame capturing.

Step 604) When the digital watermark is detected, detection of the phasesynchronization signal is tried. Since the phase synchronization signalis embedded with the modulation method (B-1), the phase synchronizationsignal can be detected if an absolute value of a correlation value withthe spreading series representing “0” is large, so that it can bedetermined whether the watermark pattern in the difference image is inphase or has a reversed phase compared with the basic watermark patternused when embedding by using polarity of plus/minus of the correlationvalue.

Step 605) If the correlation value of the phase synchronization signalwhen it is detected is plus, it is determined that they are in phase, ifit is minus, it is determined that they have reversed phases. When it isdetermined that they are in phase, the process goes to step 606, andwhen it is determined that they have the reversed phases, the processgoes to step 607.

Step 606) When it is determined that they are in phase, the detectedwatermark information is output as a detection result as it is.

Step 607) When it is determined that they have the reversed phases, bitsof the detected watermark information are reversed to output a detectionresult.

Effects of the Present Embodiment

A part of the watermark information is used as the bit reversaldetermination flag in the fourth embodiment. On the other hand, in thepresent embodiment, the phase synchronization signal for determining thephase of the watermark pattern in the difference image is multiplexedand embedded with watermark information. At this time, since the phasesynchronization signal is embedded using a modulation method that is thepattern amplitude using type and that is the polarity non-using type,the phase of the watermark pattern in the difference image can bedetermined using polarity when performing detection. Accordingly,correct watermark information can always be detected irrespective ofselection methods for selecting two frames obtained by capturing likethe fourth embodiment.

In addition, compared with the fourth embodiment, since the bit reversaldetermination flag is not used, it is not necessary to shorten bitlength of the watermark information by one bit. By the way, a patternshape using type may be used for embedding the phase synchronizationsignal as long as reversal of the difference image can be specified.

Sixth Embodiment

In this embodiment, an example is described in which it is determinedwhether the watermark pattern in the difference image is in phase withthe basic watermark pattern used when embedding, and the watermarkpattern is phase-reversed and stored as necessary.

This embodiment is the same as the first to fifth embodiments except forparts described below. In the following, for simplifying explanation,differences from the first embodiment are described.

FIG. 41 shows a configuration of a digital watermark detection unit inthe sixth embodiment of the present invention.

The digital watermark detection unit 240 of the digital watermarkdetection apparatus 200 shown in the figure includes a difference imagephase determination unit 241, a watermark information detection unit 242and a difference image storing buffer 255.

FIG. 42 is a flowchart of operation of the digital watermark detectionunit in the sixth embodiment of the present invention.

Step 701) When the difference image phase determination unit 241receives a difference image, the difference image phase determinationunit 241 determines phase of a watermark pattern in the receiveddifference image.

Step 702) The difference image phase determination unit 241 determineswhether the phase of the watermark pattern in the input difference imageis the same as or reversed from the phase of the basic watermark patternused for embedding. When they are in phase, the process goes to step703. When they are reversed, the process goes to step 704.

Step 703) When they are is in phase, the currently received differenceimage is added to the difference image storing buffer 255 as it is, andthe process goes to step 705.

Step 704) When they have reversed phases, the currently receiveddifference image is phase-reversed and the phase-reversed image is addedto the difference image storing buffer 255.

Step 705) Next, the watermark information detection unit 242 tries todetect watermark information from the added difference image stored inthe difference image storing buffer 255.

Step 706) When detection succeeds, the process goes to step 708, andwhen the detection fails, the process goes to step 707.

Step 707) When the detection fails, the process returns to a process forcapturing a next frame by the moving image input unit 210.

Step 708) When the detection succeeds, the detection result is output.

Next, the difference image phase determination unit 241 is described.

FIG. 43 is a figure for explaining processes of the difference imagephase determination unit in the sixth embodiment of the presentinvention.

When the difference image is received, the difference image phasedetermination unit 241 determines whether the phase of the watermarkpattern in the difference image is the same as or reversed from thephase of the basic watermark pattern used when embedding. Asdetermination methods, there may be a method for detecting a bitreversal determination flag described in the fourth embodiment todetermine same in-phase/reversed phase using the value, and a method fordetecting a phase synchronization signal shown in the fifth embodimentto determine in phase/reversed phase using the value. Alternatively, ina case where the watermark information is modulated using the patternamplitude using type and the polarity non-using type modulation like themodulation method (B-1) of the first embodiment, it is possible todetermine that they are in phase when a total sum of correlation valuesof the maximum absolute value detected in the detection process for eachsymbol, and to determine that they have reversed phases when the totalsum is minus.

When it is determined that the watermark pattern in the difference imageis in phase with the basic watermark pattern used when embedding, thecurrently received difference image is added to the difference imagestoring buffer 255 as it is. When it is determined that the phase isreversed, the phase of the currently received difference image isreversed and the reversed image is added to the difference image storingbuffer 255. For reversing the phase, minus/plus of pixel values of thewhole image may be reversed as to luminance components, for example. Or,if the detection subject region images used for generating the currentdifference image are X and Y, and if the current difference image isobtained by X−Y, a difference image of Y−X may be used for reversing thephase.

In addition, like the third embodiment, when synchronization ofwatermark pattern switching timing is realized using the watermarkpattern switching information used when embedding, by using a principlethat phases of adjacent watermark patterns may be reversed with eachother but phases of every other watermark pattern may be the same basedon switching timing, for example, the image may be stored whilereversing phase every other watermark pattern such that every watermarkpattern becomes the same phase when storing difference images.

According to the above-mentioned processes, difference images eachhaving a watermark pattern having a phase the same as the phase of thebasic watermark pattern are added sequentially in the difference imagestoring buffer 255 as shown in FIG. 44. By the way, “add differenceimage” means obtaining an image, having the same size, by adding pixelvalues for each point of coordinates of images.

By the way, as to storing using the before-mentioned third embodiment,the result stored in the difference image storing buffer 255 is obtainedsuch that images are added in a state in which phases that are the sameor reversed compared with the phase of the basic watermark pattern whenembedding are uniformed. By using the contrivance of the fourthembodiment or the fifth embodiment when performing detection from thedifference image storing buffer 255, the problem due to the reversedphase can be avoided.

Effects of the Present Embodiment

In the present embodiment, it is determined whether the watermarkpattern in the difference image is in phase with the basic watermarkpattern used when embedding, and the phase is reversed as necessary sothat the image is sequentially added in the difference image storingbuffer 255 always in a state of a positive phase and detection ofwatermark information is tried from the added difference image.

Accordingly, even when a watermark pattern in a difference image is weakso that the watermark pattern cannot be sufficiently detected, since aplurality of difference images are integrated and detection isperformed, detection can be performed. As shown in FIG. 44, since thewatermark pattern is added always in a form of the positive phase, awatermark pattern that is amplified by the number of times the imagesare added can be obtained in the difference image storing buffer 255 inprinciple. However, generally, since each difference of the originalmoving images is different for each timing, a difference signal of theoriginal moving images that is noise for the watermark pattern is notamplified by the number of added times. Accordingly, S/N ratio of thewatermark signal improves, and tolerance of the digital watermarkimproves compared with the first to fifth embodiments.

In addition, this tolerance improvement can be used for suppressingimage quality deterioration. That is, watermark can be detected by thedifference image storing method even though the watermark embeddingstrength is weaker than the strength for enabling detection from onedifference image like the first to fifth embodiment. Therefore, imagequality improvement can be realized.

Seventh Embodiment

In this embodiment, a method is described in which detection can besuccessfully performed even when sufficient detection performance is notobtained from one difference image.

This embodiment is the same as the first to sixth embodiments except forparts described below. In the following, for simplifying explanation,differences from the first embodiment is described.

FIG. 45 shows a configuration of the digital watermark detection unit inthe seventh embodiment of the present invention.

The digital watermark detection unit of the digital watermark detectionapparatus 200 shown in the figure includes a correlation calculationunit 243, a detection necessity determination unit 244 and a correlationvalue buffer 245.

FIG. 46 shows a flowchart of operation of the digital watermarkdetection unit in the seventh embodiment of the present invention.

Step 801) The digital watermark detection unit 240 receives a differenceimage, and obtains detection subject series from the difference image inthe same way as a normal digital watermark detection process by thecorrelation calculation unit 243 so as to obtain correlation valuesbetween the detection subject series and spreading series. By the way,at this time, if watermark information is successfully detected onlyfrom the received difference image, the digital watermark detection unit240 may output the watermark information as a detection result andterminate the process. As to the correlation value, values the number ofwhich is the same as bits of the watermark information are obtained whenusing a bit-by-bit spreading scheme like the modulation methods(A-1)-(A-4), and values whose range is the same as that of number ofsymbols X symbol values are obtained when using a symbol-by-symbolspreading scheme like the modulation methods (B-1)-(B-4).

Step 802) The correlation value group obtained by the correlationcalculation unit 243 is added in the correlation value buffer 245 foreach term.

Step 803) Next, the detection availability determination unit 244performs threshold-determination for the correlation value group storedin the correlation value buffer 245 to determine availability fordetecting watermark information.

Step 804) When the watermark information is detected, the process goesto step 806, and when the watermark is not detected, the process goes tostep 805.

Step 805) When the watermark information is not detected, the processreturns to next frame capturing by the moving image input unit 210.

Step 806) If threshold-determination is passed, it is regarded thatdetection succeeds, so that detected watermark information is formedbased on contents in the correlation value buffer 245 to output adetection result.

Next, processes on the correlation value buffer 245 are described.

FIG. 47 is a figure for explaining contents of processes on thecorrelation value buffer in the seventh embodiment of the presentinvention.

The correlation calculation unit 243 obtains the detection subjectseries from the difference image to calculate correlation between thedetection subject series and spreading series used for modulation. Whenusing the bit-by-bit spreading scheme like the modulation methods(A-1)-(A-4), values the number of which is the same as bits of thewatermark information are obtained, when using the symbol − by symbolspreading scheme like the modulation methods (B-1)-(B-4), values whoserange is the same as that of the number of symbols X symbol values areobtained (FIG. 47 shows an example in which 1 symbol equals to 1 bit andthere are four symbols each taking two values “0” or “1”.)

As to a method like the modulation method (A-1) in which bits may bereversed according to a phase of the difference, correlation iscalculated after aligning phases using the method described in thefourth embodiment or the fifth embodiment. In addition, in a case ofmodulation methods (B-1)-(B-4), when the phase of the difference isreversed, an absolute value of the correlation value is calculated sincethe correlation value becomes minus when the phase of the difference isreversed. The correlation value group obtained by the correlationcalculation unit 243 is added in the correlation value buffer 245 foreach term of each correlation value.

In addition, like the third embodiment, in a case where synchronizationof watermark pattern switching timing can be performed using watermarkpattern switching information used when embedding, by using a principlethat switching between adjacent watermark patterns forms reversed phaseswith each other but switching for every other pattern forms same phases,for example, polarity of the correlation values can be always alightedby reversing the polarity of the correlation value for every other imageand the image is added and stored.

FIGS. 48A and 48B are figures for explaining contents of processes onthe correlation value buffer in the seventh embodiment of the presentinvention.

The detection availability determination unit 244 performs thresholddetermination for the correlation values stored in the correlationbuffer 245 to determine whether detection succeeds. For the case of thebit-by-bit spreading scheme like the modulation methods (A-1)-(A-4), itmay be determined that detection succeeds if all of absolute values ofcorrelation values for each bit position are equal to or greater than athreshold. Or, threshold determination may be performed for a result ofcalculating sum of absolute values of correlation values for each bitposition as shown in FIGS. 48A and 48B. In addition, as to thesymbol-by-symbol spreading scheme like the modulation methods(B-1)-(B-4), it is determined that detection succeeds if all of maximumvalues of correlation values of each symbol position are equal to orgreater than a threshold, for example. By the way, the threshold valuemay be changed according to the number of times of addition to thecorrelation value storing buffer 245, or each term value in thecorrelation value storing buffer 245 may be divided by the number oftimes of addition before performing the threshold determination asnecessary.

By the way, as to a storing method using the third embodiment, theproblem due to polarity reversal can be avoided by using the contrivancein the fourth or fifth embodiment for the result stored in thecorrelation value buffer 245.

Effects of the Present Embodiment

In this embodiment, detection success and failure is determined based onthe result obtained by adding and storing correlation values calculatedwhen performing detection from the difference image. Accordingly, evenwhen detection is difficult by performing detection trial once,detection can be realized by integrating detection results from aplurality of difference images so that detection performance improves.

Eighth Embodiment

This embodiment is the same as the first to seventh embodiments exceptfor parts described below. In the following, for simplifyingexplanation, differences from the first embodiment are described.

FIG. 49 shows a configuration of the watermark pattern generation unitin the eighth embodiment of the present invention. The watermark patterngeneration unit 120 of the digital watermark apparatus 100 shown in thefigure includes the basic watermark pattern generation unit 121, thewatermark pattern switching unit 122 and a watermark informationdividing unit 123.

FIG. 50 shows a flowchart of operation of the watermark patterngeneration unit in the eighth embodiment of the present invention.

Step 901) When the watermark pattern generation unit 120 receiveswatermark information, watermark pattern switching information, and aframe display time, the watermark information dividing unit 123subdivides the watermark information into data block information thatare bit information shorter than the watermark information.

Step 902) The watermark information dividing unit 123 refers to theframe display time so as to send, to the basic watermark patterngeneration unit 121, a data block ID indicating what number a data blockto be embedded in the current frame image is and data block informationthat is bit information of the data block corresponding to the datablock ID.

Step 903) The basic watermark pattern generation unit 121 generates abasic watermark pattern from the data block ID and the data blockinformation.

Step 904) Finally, the watermark pattern switching unit 122 determinesnecessarily of reversing the phase of the basic watermark pattern basedon relationship between the frame display time and the watermark patternswitching information.

Step 905) When it is necessary to reverse the phase of the basicwatermark pattern, the process goes to step 906, and when it is notnecessary, the process goes to step 907.

Step 906) When it is necessary to reverse the phase, the phase of thebasic watermark pattern is changed so that the phase changed pattern isoutput as a watermark pattern, and the process is terminated.

Step 907) The basic watermark pattern is output and the process isterminated.

Next, the watermark information dividing unit 123 is described indetail.

FIG. 51 is a figure for explaining processes of the watermarkinformation dividing unit in the eighth embodiment of the presentinvention.

In the example shown in the figure, it is assumed that the watermarkswitching information indicates “reverse every 1/10 second”, the framerate of the original moving image data is 30 frames per second, and thatthe watermark information is 16 bits. The watermark information dividingunit 123 divides the watermark information into data blocks each havinga predetermined bit length. In the example shown in FIG. 51, watermarkinformation of 16 bits is divided into four four-bit data blocks.

In addition, data block IDs (a, b, c and d in FIG. 51) each indicatingwhat number the data block is are simultaneously obtained. Then, thewatermark information dividing unit 123 selects a data block to beembedded into the current frame, and sends selected data blockinformation and the data block ID to the basic watermark patterngeneration unit 121. The example of FIG. 51 shows contents of processthat is “switch data block at intervals of one period of watermarkpattern reversal ( 6/30 seconds)”, and the data block is switched every6 frames in the example.

By the way, when the data block reaches the last data block, the processreturns to the top so as to perform selection cyclically. In addition,when the last data block is selected, also termination informationindicating termination of the watermark information may be sent to thebasic watermark pattern generation unit 121.

Although FIG. 51 shows an example for sequentially switching the datablock every one period of watermark pattern reversal, various variationscan be adopted such as switching every three reversal periods, everydata block switching period other than the reversal period, or selectingrandomly instead of sequentially switching data block.

Next, the basic watermark pattern generation unit 121 is described.

FIGS. 52A and 52B are figures for explaining processes of the basicwatermark pattern generation unit in the eighth embodiment of thepresent invention.

Different from the first to sixth embodiments, the basic watermarkpattern generation unit 121 generates a basic watermark pattern bymodulating, instead of watermark information itself, the data blockobtained by dividing the watermark information and the data block IDindicating order of the data block.

When receiving termination information indicating the last data blockfrom the watermark information dividing unit 123, the basic watermarkpattern generation unit 121 may add “1” such that termination of datablock can be detected when performing detection, but it is not necessarywhen always using fixed number of data blocks, for example, using only 4data blocks.

In addition, like the fourth to sixth embodiments, information embeddingmay be performed simultaneously for determining the phase of thedifference image when performing detection.

FIG. 53 shows a configuration of the digital watermark detectionapparatus in the eighth embodiment of the present invention.

A difference between the digital watermark detection apparatus 200Dshown in the figure and the apparatus 200A in the first embodiment isprocesses performed after the digital watermark detection unit 240.

FIG. 54 shows a flowchart of operation of the digital watermarkdetection apparatus in the eighth embodiment of the present invention.

Step 1001) The moving image input unit 210 sequentially obtains a frameimage by capturing the moving images.

Step 1002) A detection subject region extraction unit 220 extracts adetection subject region image in a captured frame image.

Step 1003) The difference image generation unit 230 generates adifference image between a currently obtained detection subject regionimage and a previously obtained detection subject region image.

Step 1004) The digital watermark detection unit 240 tries to detectembedded information by performing digital watermark detection trialfrom the difference image in the same way as the first embodiment.

Step 1005) When detection succeeds, the process goes to step 1006. Whendetection does not succeed, the process goes to step 1001.

Step 1006) Different from the first embodiment, when detection succeeds,since the detected information is not the watermark information but isdata block information that is a part of the watermark information and acorresponding data block ID, these pieces of information are sent to thedetection data block storing unit 280. The detection data block storingunit 280 sets a data value of a detected watermark information buffer ata position indicated by the currently detected data block ID to be thevalue of the detected data block information.

Step 1007) When all data blocks in the detected watermark informationbuffer are successfully detected, the process goes to step 1008. Whenthe digital watermark detection unit 240 fails digital watermarkdetection from the difference image, or when there remains a data blockthat is not yet buffered in the detected data block storing unit 280,the process returns to capturing in the moving image input unit 210 tocontinue the detection process.

Step 1008) The value in the detected watermark information is regardedas detected watermark information and the detection result is output.

FIG. 55 shows a configuration of the detected data block storing unit inthe eighth embodiment of the present invention. The detected data blockstoring unit 280 shown in the figure includes a detected data blockbuffering unit 281, a detection complete check unit 282 and a detectedwatermark information buffer 283.

Operation of the detected data block storing unit 280 is described.

FIG. 56 shows a flowchart of operation of the detected data blockstoring unit in the eight embodiment of the present invention. FIGS. 57and 58 show contents of processes of the detected data block storingunit.

Step 1101) The detected data block storing unit 280 receives a detecteddata block ID and detected data block information.

Step 1102) First, the detected watermark data block buffering unit 281sets a data value at a position indicated by the received detected datablock ID in the detected watermark information buffer 283 to be thevalue of the received detected data block information.

As shown in FIG. 57, the detected watermark information buffer 283 is ina state in which no data block is detected for every data block ID in aninitial state, and the state sequentially changes from the undetectedstate to detected state each time when a detected data block ID anddetected data block information are received. By the way, when a datablock ID that is already detected is received, the data value of thecorresponding data block ID in the detected watermark information buffer283 may be overwritten with the detected data block information, ornothing may be performed without performing overwriting.

FIG. 57 shows contents of processes in a case where the number of datablocks are predetermined. In a case where a termination flag of datablocks is embedded when embedding, it cannot be ascertained how manydata blocks are in the initial state as shown in FIG. 58. However, byreceiving a data block ID including a termination flag, total number ofdata blocks can be ascertained. Thus, an undetected data block can bespecified. After that, processes the same as those when the number ofdata blocks is predetermined can be performed. By using the terminationinformation, it becomes possible to deal with embedding/detection ofwatermark information of an arbitrary length.

Step 1103) Next, the detection completion check unit 282 determineswhether all data blocks in the detected watermark information buffer 283are “detected”, and if all data blocks are detected, the process goes tostep 1104, and when an undetected data block is remained, the processgoes to step 1105.

Step 1104) When all data blocks are detected, the data values in thedetected watermark information buffer 283 are regarded as detectedwatermark information, and the detection result is output.

Step 1105) When an undetected data block remains, the process returns toframe capturing by the moving image input unit 210 so as to continuedetection trial.

By the way, in the present embodiment, although differences from thefirst embodiment are mainly described, it is obvious that thecontrivance of the present embodiment can be easily carried out bycombining with contrivances in the second to seventh embodiments. Forexample, it is obvious that detection performance improves by storingdifference images for each data block like the sixth embodiment, and bystoring correlation values like the seventh embodiment.

Effects of this Embodiment

In the present embodiment, watermark information is divided into datablocks and watermark is embedded while temporally switching data blocks.In a detection side, detected data blocks are sequentially stored, andwhen all data blocks are detected, watermark detection is regarded to besucceeded. Accordingly, it becomes possible to performembedding/detection of watermark information having an informationlength longer than an embedding information length limit of a watermarkalgorithm for one difference image, or tolerance improvement can berealized by decreasing an information length embedded into each frame.

In addition, by embedding/detecting termination information, informationof an arbitrary length can be dealt with, so that an application thatrequires longer information can be realized, for example. Thus, usecoverage widens.

Ninth Embodiment

In this embodiment, an example is described in which the data block IDis embedded by multiplexing it into watermark information usingmodulation that is the pattern amplitude using type and that is apolarity non-using type.

This embodiment is the same as the eighth embodiment except for partsdescribed below.

Contents of processes of the basic watermark pattern generation unit 121in the digital watermark embedding apparatus 100 are described.

FIG. 59 is a figure for explaining an example of processes of the basicwatermark pattern generation unit in the ninth embodiment of the presentinvention.

In the present embodiment, as to the data block information and the datablock ID that are inputs for a basic watermark pattern, first, in thesame way as the fifth embodiment, the data block information ismodulated to obtain a basic watermark pattern (kou).

Next, as to the data block ID, instead of the phase synchronizationsignal (always same value) of the fifth embodiment, a data block IDhaving multiple values (including data block termination information asnecessary) is modulated using the modulation method (B-1) to obtain abasic watermark pattern (otsu).

Finally, the basic watermark pattern (kou) and the basic watermarkpattern (otsu) are superimposed and combined to obtain a basic watermarkpattern.

FIG. 60 shows a flowchart of operation of the digital watermarkdetection unit in the ninth embodiment of the present invention.

Step 1201) The digital watermark detection unit 240 receives adifference image, and tries to detect data block information from thedifference image in the same way as the eighth embodiment.

Step 1202) When the data block information is detected, the process goesto step 1204, and when it is not detected, the process goes to step1203.

Step 1203) The process returns to next frame capturing by the movingimage input unit 210.

Step 1204) When the data block information is detected, it is tried todetect the data block ID next.

Step 1205) Since the data block ID is embedded by the modulation method(B-2), the data block ID can be detected using an absolute value ofcorrelation calculation first, and it can be determined whether thephase of the watermark pattern in the difference image is the same as orreversed from the phase of the basic watermark pattern used whenembedding using plus/minus polarity of the correlation value at thattime. When the correlation value of the data block ID when detecting isplus, it is determined that the phase is a positive phase, and when thecorrelation value is minus, it is determined that the phase is areversed phase. When it is determined that the phase is the positivephase, the process goes to step 1207, and when it is determined that thephase is the reversed phase, the process goes to step 1206.

Step 1206) When the phase is the reversed phase, bits of the detecteddata block information are reversed and it is output with the data blockID.

Step 1207) When it is determined that the phase is the positive phase,the detected data block information is output as it is with the datablock ID.

Like the eighth embodiment, it is obvious to carry out contrivances ofthis embodiment by combining with contrivances of the first to seventhembodiments. For example, it is obvious that detection performanceimproves by storing difference images for each data block like the sixthembodiment, and by storing correlation values like the seventhembodiment.

Effects of the Present Embodiment

In this embodiment, the data block ID is multiplexed into the watermarkinformation using modulation that is the pattern amplitude using typeand that is the polarity non-using type to perform embedding.Accordingly, information length of data block information embedded ineach frame can be made long.

In addition, since phase of the watermark pattern can be determinedusing polarity of correlation when detecting the data block ID, it isnot necessary to prepare the flag for determining bit reversal, and itcontributes to extension of information length. That is, by using thepolarity non-using type modulation for the data block ID, it becomespossible to determine the phase of the watermark pattern simultaneouslywith detection of the data block ID. Thus, the technique serves a dualpurpose.

By the way, the pattern shape using type that can specify reversal ofthe difference image can be used for data block ID embedding.

Tenth Embodiment

In the present embodiment, an example for storing difference images inwhich phases are aligned is described.

The embodiment is the same as the first to ninth embodiments except forparts described below. In the following, for simplifying explanation,differences from the eighth embodiment are described.

The digital watermark detection unit 240 in the digital watermarkdetection apparatus 200 in this embodiment is described.

FIG. 61 shows a configuration of the digital watermark detection unit inthe tenth embodiment of the present invention.

The digital watermark detection unit 240 shown in the figure includes adifference image phase determination unit 241, a data block ID detectionunit 246, a data block information detection unit 247, and differenceimage storing buffers 248 for data block ID=n.

FIG. 62 shows a flowchart of operation of the digital watermarkdetection unit in the tenth embodiment of the present invention.

Step 1301) When receiving the difference image, the digital watermarkdetection unit 240 tries to detect the data block ID by the data blockID detection unit 246.

Step 1302) Next, the difference image phase determination unit 241detects the phase of the watermark pattern in the difference image inthe same way as the sixth embodiment.

Step 1303) When the phase is positive, the process of the differenceimage phase determination unit 241 goes to step 1304, and when the phaseis reversed, the process goes to step 1305.

Step 1304) When the phase is positive, the current difference image isadded and stored in a difference image buffer 248 corresponding todetected data block ID, and the process goes to step 1306.

Step 1305) When the phase is reversed, the difference image is changedsuch that the phase becomes positive, and is stored in the differenceimage buffer 248 corresponding to the detected data block ID. As shownin FIG. 61, the digital watermark detection unit 240 includes differenceimage buffers 248 the number of which is the same as a number of kindsof data block IDs.

Step 1306) Next, the data block information detection unit 247 tries todetect data block information from the difference image storing buffer248 corresponding to the currently detected data block ID.

Step 1307) When detection succeeds, the process goes to step 1308. Whendetection does not succeed, the process goes to step 1309.

Step 1308) When the detection succeeds, the current data block ID andthe data block information are output.

Step 1309) When the detection does not succeed, the process returns tocapturing process by the moving image input unit 210.

Effects of the Present Embodiment

According to the present embodiment, when enlarging the length of thewatermark information using the data blocks, tolerance can be alsoimproved by storing difference images at the same time so that detectionperformance improves.

Eleventh Embodiment

In this embodiment, an example for storing correlation values isdescribed.

This embodiment is the same as the first to tenth embodiments except forparts described below. In the following, differences from the eighthembodiment are described for simplifying explanation.

FIG. 63 shows a configuration of the digital watermark detection unit inthe eleventh embodiment of the present invention.

The digital watermark detection unit 240 shown in the figure includes adata block ID detection unit 246, a detection availability determinationunit 244, a correlation calculation unit 249, correlation buffers 251for data block ID=n. As shown in the figure, the digital watermarkdetection unit 240 includes correlation value buffers for data blockID=n the number of which is the same as the number of kinds of datablock IDs.

Operation of the digital watermark detection unit 240 configured asmentioned above is described below.

FIG. 64 is a flowchart showing operation of the digital watermarkdetection unit in the eleventh embodiment of the present invention.

Step 1401) When receiving the difference image, the digital watermarkdetection unit 240 tries to detect the data block ID by the data blockID detection unit 246.

Step 1402) Next, the correlation calculation unit 249 obtains acorrelation value calculated by performing data block informationdetection trial.

Step 1403) The obtained correlation value is added and stored in acorrelation value buffer 251 corresponding to the detected data blockID. By the way, at this time, when detection of the data blockinformation succeeds only from the received difference image, it may beoutput with the data block ID to terminate the process.

Step 1404) Next, as to the correlation value buffer 251 corresponding tothe currently detected data block ID, the detection availabilitydetermination unit 244 determines detection possibility for data blockinformation by performing threshold determination in the same way as theseventh embodiment.

Step 1405) When it is determined that detection is possible, the processgoes to step 1406, and when the detection does not succeed, the processgoes to step 1407.

Step 1406) When it is determined that detection is possible, data blockinformation is formed from values in the current correlation valuebuffer 251, and is output with the current data block ID.

Step 1407) When detection does not succeed, the process returns tocapturing process by the moving image input unit 210.

Effects of this Embodiment

According to the present embodiment, when enlarging the watermarkinformation length using the data block, tolerance improvement by thecorrelation value buffer 251 can be also realized at the same time, sothat detection performance improves.

Twelfth Embodiment

In the present embodiment, an example is described in which status ofdigital watermark detection process is fed back and output in real time.

This embodiment is the same as the first to eleventh embodiments exceptfor parts described below. In the following, differences from the firstembodiment are described for simplifying explanation.

FIG. 65 shows a configuration of a digital watermark detection apparatusof the twelfth embodiment of the present invention. A difference betweenthe digital watermark detection apparatus 200E shown in the figure andthe first embodiment is that the digital watermark detection apparatus200E includes a feedback output unit 265 that receives a currentlycaptured frame image, current detection subject region extraction statusinformation and detection status information for current digitalwatermark detection as needed, and generates and outputs feedbackinformation in parallel with performing digital watermark detectioncorresponding to the first embodiment.

FIG. 66 is a flowchart of operation of the digital watermark detectionapparatus in the twelfth embodiment of the present invention.

In the figure, since steps 1501-1506 are the same as steps 201-206 inFIG. 20, the steps are not described. These steps are performed inparallel with the following process of step 1507.

Step 1507) The feedback output unit 265 receives a currently capturedframe image, current detection subject region extraction statusinformation and detection status information for current digitalwatermark detection as needed, and generates and outputs feedbackinformation in parallel with performing digital watermark detectioncorresponding to the first embodiment.

FIGS. 67, 68 and 69 show examples of the feedback output of the digitalwatermark detection apparatus in the twelfth embodiment of the presentinvention.

A preview screen is provided for drawing, in real time, the capturedframe image obtained by the moving image input unit 210 of the digitalwatermark detection apparatus 200E, and the detection subject regionobtained by the detection subject region extraction unit 220 issynthesized on the preview screen (a in FIG. 67), so that detectionstrength obtained by the digital watermark detection unit 240 such as asize of the correlation value when performing demodulation isgraphically displayed (b in FIG. 67) or voice is output by changing toneor volume according to the detection strength.

In addition, when using the data block, detection status for data blocksthat are buffered until now are synthesized as shown in FIG. 68.

Alternatively, as shown in FIG. 69, when degree of plane projectiondistortion caused by image-taking angle is large, or when pixel area inthe detection subject region in the captured frame image is too smallsince the image is taken from a distant position so that detection ofdigital watermark is difficult, a message indicating that adjustment ofthe angle or zooming up is necessary is displayed to urge the user toperform adjustment.

Effects of the Present Embodiment

In the present embodiment, interactivity increases by feeding back andoutputting the status of the digital watermark detection process in realtime so that convenience improves. Especially, by urging the user tomake a status in which detection becomes easy, improvement of detectionperformance can be expected.

Thirteenth Embodiment

This embodiment is the same as the first to twelfth embodiments exceptfor parts described below. In the following, differences from the firstembodiment are described to simplify explanation.

Processes of the watermark pattern superimposing unit 130 in the digitalwatermark embedding apparatus 100 of this embodiment is described.

FIG. 70 is a figure for explaining processes of the watermark patternsuperimposing unit in the thirteenth embodiment of the presentinvention.

The watermark pattern superimposing unit 130 receives a frame image anda watermark pattern generated based on the frame display time. Thewatermark pattern superimposing unit 130 changes the size of thewatermark pattern to a predetermined size that is equal to or smallerthan that of the frame image. After that, the watermark patternsuperimposing unit 130 adds the watermark pattern onto a central area ofthe frame image to obtain a watermark embedded frame image.

Next, the digital watermark detection apparatus 200F in this embodimentis described.

FIG. 71 shows a configuration of the digital watermark detectionapparatus in the thirteenth embodiment of the present invention.

The digital watermark detection apparatus 200F shown in the figureincludes a moving image input unit 210, a difference image generationunit 230, a detection subject region extraction unit 220, a digitalwatermark detection unit 240, and a frame image buffer 301, wherein themoving image input unit 210 receives data obtained by real-timevideo-capturing of analog moving images displayed on a TV and the like,or receives digital moving images that are MPEG-encoded.

Operation of the above-mentioned configuration is described.

FIG. 72 is a flowchart of operation of the digital watermark detectionapparatus in the thirteenth embodiment of the present invention.

Step 1501) The moving image input unit 210 inputs the analog or digitalmoving images to obtain a frame image sequentially. When inputting theanalog moving images, camera, scanner, or analog video signals are inputso as to obtain the frame image. In the case of digital moving images,the frame image is obtained by performing decoding processes and thelike.

Step 1502) Next, the difference image generation unit 230 calculate adifference between the currently obtained frame image and a previouslyobtained frame image stored in the frame image buffer 301 to generate adifference frame image.

Step 1503) In addition, in preparation for next detection trial, thecurrent frame image is buffered in the frame image buffer 301.

Step 1504) Next, the detection subject region extraction unit 220extracts a detection subject region that becomes a subject of watermarkdetection from the difference frame image to obtain a detection subjectregion image.

Step 1505) Next, the digital watermark detection unit 240 tries todetect digital watermark from the detection subject region image andoutputs a detection result.

Step 1506) When digital watermark detection does not succeed, the movingimage input unit 210 obtains a next frame image to repeat theabove-mentioned process sequentially.

Next, processes of the difference image generation unit 230 aredescribed in detail.

FIG. 73 is a figure for explaining processes of the difference imagegeneration unit in the thirteenth embodiment of the present invention.

The difference image generation unit 230 obtains a difference betweenthe currently obtained frame image and a previously obtained frame imagestored in the frame image buffer 301 to generate a difference frameimage. Here, it is assumed that there is little camera movement whentaking the two frame images. As shown in FIG. 73, in the differenceframe image, a part in which difference is large caused by watermarkpattern switching can be distinguished from a background part in whichthere is no change.

Next, processes of the detection subject region extraction unit 220 aredescribed in detail.

FIG. 74 is a figure for explaining processes of the detection subjectregion extraction unit in the thirteenth embodiment of the presentinvention.

The detection subject region extraction unit 220 receives the differenceframe image and extracts the detection subject region that is a subjectfor watermark detection to obtain the detection subject region image. Asshown in FIG. 74, the detection subject region extraction unit 220changes values of the difference frame image into absolute values,detects distortion parameters of the detection subject region using anedge detection method of the document 3 so as to perform distortioncorrection of the original difference frame image before being changedto the absolute values and normalize the size to obtain the detectionsubject region image. Since the detection subject region image obtainedin this way is the same as the basic watermark pattern used forembedding excluding effects due to noise and the like, it is obviousthat detection of digital watermark becomes possible from contents ofthe first to twelfth embodiments.

Also, it is obvious that the present embodiment can be carried out byarbitrarily combining with each contrivance of the first to twelfthembodiments.

Effects of the Present Embodiment

Effects of the present embodiment is described with reference to FIGS.75-77.

First, in digital watermark embedding in the first to twelfthembodiments, a watermark pattern having a size the same as the framesize of the received moving image is superimposed. In addition, whendetecting a digital watermark, the detection subject region is extractedby detecting an edge of a display area of a TV receiver in a framecaptured by a camera. However, as shown in FIG. 75, since it can beconsidered that environment such as a shape of the TV receiver forreproducing the moving images is not constant, when a display screenarea is correctly extracted as a detection subject region, digitalwatermark detection succeeds as shown in FIG. 75A. But, there may be acase in which the detection subject region is not correctly extractedwhen an outside of the TV receiver is detected as the edge as shown inthe example of FIG. 75B, or when a frame-like design attached outsidethe display screen of the TV receiver is detected as shown in theexample of FIG. 75C.

In such cases, the watermark pattern becomes a contracted shape in theinside of the difference image on the difference image and blockdividing cannot be performed correctly. Therefore, digital watermarkdetection fails in the first to twelfth methods.

In addition, as shown in FIG. 76, an example is considered to displayoriginal moving image that is a wide video having an aspect ratio of16:9 on a monitor of an aspect ratio of 4:3. In this case, as displaymethods that are usually used in general, there are a method for cuttingout left and right ends of the wide screen to display the cut out image,and a method for contracting the wide screen of 16:9 so as to fit itinto the 4:3 screen. When the image is displayed in such a way, eventhough the edge of the display screen is correctly detected by detectionsubject region extraction, since a part of the displayed image itself iscut out or useless white space is added, digital watermark fails in thesame way as the example of FIG. 75. As to the problem in that the endsof the screen are cut out, in addition to the case of the wide image, itcan be considered that detection of digital watermark becomes difficultdue to so-called over scan in which ends of video signals are notdisplayed when displaying video on a cathode-ray tube so that a part ofthe video signals is cut out.

The present embodiment is for solving the problems, and the size of thewatermark pattern is decreased to a size smaller than a size of theframe image so that it is superimposed onto a center area of the frameimage. Accordingly, for example, in anticipation of lack of the frameimage when displaying a wide screen by the ratio 4:3, the watermarkpattern can be superimposed only on an region that is surly displayed.However, if nothing is done, the detection subject region does not agreewith the watermark pattern region when performing detection. Therefore,in this embodiment, the difference frame image between captured frameimages is generated as shown in FIG. 73, and the detection subjectregion is extracted from the difference frame image as shown in FIG. 74.When it is assumed that there is little movement when taking images witha camera, (absolute value of) difference only in the detection subjectregion becomes large by switching watermark patterns in the differenceframe image. Accordingly, the detection subject region is extracted by arectangular recognition technique using edges and the like, so thatdigital watermark detection becomes possible as shown in FIGS. 77 and78.

Fourteenth Embodiment

The present embodiment is the same as the thirteenth embodiment exceptfor parts described below.

FIG. 79 is a figure for explaining processes of the basic watermarkpattern generation unit of the fourteenth embodiment of the presetinvention.

The basic watermark pattern generation unit 121 of the watermark patterngeneration unit 120 in this embodiment adds a frame to the basicwatermark pattern generated in the same way as the first to twelfthembodiments using plus/minus pixel values so as to regard the frameadded pattern as a basic watermark pattern.

FIG. 80 is a figure for explaining processes of the detection subjectregion extraction unit in the fourteenth embodiment of the presentinvention.

The detection subject region extraction unit 220 receives a differenceframe image, extracts the detection subject region that becomes asubject for watermark detection to obtain the detection subject regionimage. As shown in FIG. 80, the detection subject region extraction unit220 detects distortion parameters of the detection subject region usingan edge detection method of the document 3 for the difference frameimage so as to perform distortion correction and normalize the size toobtain the detection subject region image. Since the detection subjectregion image obtained in this way is the same as the basic watermarkpattern used for embedding excluding effects due to noise and the like,it is obvious that detection of digital watermark becomes possible fromcontents of the first to twelfth embodiments.

Also, it is obvious that the present embodiment can be carried out byarbitrarily combining with each contrivance of the first to thirteenthembodiments.

Effects of the Present Embodiment

Effects of the present embodiment are described using FIGS. 81A and 81B.In the detection subject region extraction process in the thirteenthembodiment, region extraction is performed by performing edge detectionusing absolute values of the difference frame image. At this time, sinceabsolute values of the difference frame image is used, contrast of theedge part has a difference twice the watermark pattern amplitude whenembedding disregarding effects due to noise.

In this embodiment, a frame having plus/minus pixel values is added tothe basic watermark pattern of the first to twelfth embodiments. Whenextracting a detection subject region, region extraction is performed byperforming edge recognition from the difference frame image as it is.Therefore, contrast of edge of the edge part becomes four times thewatermark pattern amplitude for embedding, so that edge detection can beeasily performed compared with the thirteenth embodiment. Therefore, thedetection subject region can be extracted more surly with higherreliability so as to be able to realize improvement of digital watermarkdetection performance.

Fifteenth Embodiment

The present embodiment is the same as the fourteenth embodiment exceptfor parts described below.

Processes of the basic watermark pattern generation unit 121 in thewatermark pattern generation unit 120 in the digital watermark embeddingapparatus of the present embodiment are described.

FIG. 82 is a figure for explaining processes of the basic watermarkpattern generation unit of the fifteenth embodiment of the presentinvention.

The basic watermark pattern generation unit 121 in the presentembodiment adds a marker for position adjustment to the basic watermarkpattern generated in the same way as the first to twelfth embodimentsusing plus/minus pixel values so as to regard the marker added patternas a basic watermark pattern. The frame in the fourteenth embodiment canbe also considered to be an example of a marker for position adjustment.In FIG. 82, a marker for position adjustment for finding corners used intwo-dimensional code such as QR code (registered trademark) is added asthe positioning marker. By the way, the basic watermark pattern of FIG.82 is one obtained by changing pixel values of the QR code (registeredtrademark) to plus/minus. In this way, it is no problem to usetwo-dimensional code itself as the basic watermark pattern.

Next, the detection subject region extraction unit 220 in the presentembodiment is described.

FIG. 83 is a figure for explaining processes of the detection subjectregion extraction unit in the fifteenth embodiment of the presentinvention.

The detection subject region extraction unit 220 receives the differenceframe image and extracts the detection subject region that is a subjectof digital watermark detection in the difference frame image to obtain adetection subject region image. In FIG. 83, the detection subject regionis extracted from the difference frame image by a method similar to thatfor detecting a positioning marker used for reading normal QR code(registered trademark).

By the way, when the two-dimensional code is used as the basic watermarkpattern like this embodiment, it is obvious that the digital watermarkdetection unit 240 can detect watermark information without problemconsidering that the digital watermark detection unit 240 performstwo-dimensional code reading process. In addition, also, it is obviousthat the present embodiment can be carried out by arbitrarily combiningwith each contrivance of the first to twelfth embodiments.

Effects of the Present Embodiment

In the present embodiment, an example is shown in which the positioningmarker, which was the frame in the fourteenth embodiment, is used as amarker for specifying the corners. Also in this embodiment, like thefourteenth embodiment, contrast difference between bright and dark inthe difference frame image becomes four times the watermark patternamplitude used when embedding so that detection performance improves.

In addition, according to the present embodiment, it can be understoodthat the existing two-dimensional code pattern can be easily diverted asa method for digital watermark generation. Accordingly, cost reductioneffect can be obtained by using existing components as a part of thedigital watermark embedding/detection apparatuses

Sixteenth Embodiment

In the present embodiment, an example is described in which a detectionsubject region is extracted from a difference image between featureregion images for which distortion of the feature region has beencorrected and size normalization has been performed.

The present embodiment is the same as the thirteenth to fifteenthembodiments except for parts described below.

The digital watermark detection apparatus of the present embodiment isdescribed.

FIG. 84 shows a configuration of the digital watermark detectionapparatus in the sixteenth embodiment of the present invention. Thedigital watermark detection apparatus 200G shown in the figure includesa moving image input unit 210, a feature region extraction unit 290, adifference image generation unit 230, a detection subject regionextraction unit 220, a digital watermark detection unit 240, and afeature region image buffer 302.

Operation of the digital watermark detection apparatus configured asmentioned above is described.

FIG. 85 is a flowchart of operation of the digital watermark detectionapparatus in the sixteenth embodiment of the present invention.

Step 1601) The moving image input unit 210 receives analog moving imagesdisplayed on a TV that are video-captured in real time by a camera orreceives MPEG encoded digital moving images, and obtains a frame imagesequentially. For inputting the analog moving images, camera signals,scanner signals or analog video signals are received to obtain the frameimage. For digital moving images, the frame image is obtained byperforming decoding process and the like.

Step 1602) Next, the feature region extraction unit 290 extracts afeature region in the frame image. For extracting the feature region,the technique such as rectangular area detection by edge recognitionshown in the document 3 is used. Distortion caused by camera takingangle and the like is corrected for the extracted feature region, andafter the size is normalized, it is output as a feature regionextraction image.

Step 1603) Next, the difference image generation unit 230 obtains adifference between the currently obtained feature region image and apreviously obtained feature region image stored in the feature regionimage buffer 302 to generate a feature region difference image.

Step 1604) In addition, in preparation for next detection trial, thecurrent feature region image is buffered in the feature region imagebuffer 302.

Step 1605) Next, the detection subject region extraction unit 220extracts a detection subject region that is a subject for watermarkdetection from the feature region difference image to obtain a detectionsubject region image.

Step 1606) Next, the digital watermark detection unit 240 tries todetect digital watermark from the detection subject region image tooutput a detection result.

Step 1607) When detection of digital watermark does not succeed, themoving image input unit 210 obtains a next frame image to sequentiallyrepeat the above-mentioned process.

Next, contents of processes of the feature region extraction unit 290are described.

FIGS. 86-90 are figures for explaining processes of the featureextraction unit in the sixteenth embodiment of the present invention.

The feature region extraction unit 290 receives a frame image to extracta feature region in the frame image. For extracting the feature region,a technique such as rectangular area detection by edge recognitiondisclosed in the document 3 is used. Distortion caused by camera takingangle is corrected for the extracted feature region, and the size isnormalized so that it is output as a feature region extracted image. Byusing the rectangular area extraction technique like the document 3, adisplay region of a TV is detected as the feature region as shown inFIGS. 86 and 90, an outside of a body of a TV is detected as the featureregion as shown in FIGS. 87 and 89, or a frame like design outside ofthe display screen of the TV is detected as the feature region, oralthough not shown in the figure, a background rectangular region andthe like existing behind the TV is detected as the feature region. So,there may be various cases as to the region extracted as the featureregion, but an important point is that the feature region imagesobtained by correcting distortion and normalizing the size can beobtained to be in a same status against picture-taking angle and thelike that changes every second according to capturing time as shown inFIGS. 86-90 if feature regions detected from each frame image aresubstantially the same irrespective of difference of picture-takingangles when taking a same subject.

As long as the condition that “if feature regions detected from eachframe image are substantially the same irrespective of difference ofpicture-taking angles when taking a same subject” is satisfied, thefeature region may be detected in any state such as the examples ofFIGS. 86 and 90. The feature region extraction method is not limited tothe rectangular area detection using edge line shown in the document 3,and any method can be used as long as the method is for stablyextracting the feature region from frame images and satisfies thecondition.

Next, contents of processes of the difference image generation unit 230are described.

FIGS. 91-93 are figures for explaining processes of the difference imagegeneration unit in the sixteenth embodiment of the present invention.

The difference image generation unit 230 obtains a difference between acurrently obtained feature region image and a previously obtainedfeature region image stored in the feature region image buffer 302 togenerate a feature region difference image. In addition, in preparationfor next detection trial, the current feature region image is bufferedin the feature region image buffer 302. As shown in FIGS. 91-93,difference of distortion caused by camera movement between capturedframes is absorbed by the feature region extraction process. Thus, as tothe feature region difference image obtained as time-difference offeature region images, change amount of difference in a part caused bywatermark pattern switching is large, and there is little difference inother parts. Accordingly, like the detection subject region extractionprocess in the thirteenth to fifteenth embodiments, digital watermarkinformation can be detected by trying to detect digital watermark afterextracting the detection subject region from the feature regiondifference image and correcting distortion and the like.

Effects of the Present Embodiment

In this embodiment, in addition to the effects of the thirteenth tofifteenth embodiments, digital watermark detection becomes possiblewithout the presupposed limitation that “when there is little cameramovement when taking images”. The reason is that distortion is correctedafter extracting the feature regions for each captured frame image toabsorb difference of distortion between frames so that subtraction forwatermark pattern parts is surly performed without displacement insubsequent difference image generation. Accordingly, digital watermarkdetection performance improves without receiving effects of handmovement and the like caused when performing image capturing from a TVby holding a camera-equipped mobile-phone with one hand, for example.

Seventeenth Embodiment

In the present embodiment, an example is described for extracting afeature region/detection subject region by searching only a neighborhoodof a position where digital watermark detection status is good.

This embodiment is the same as the thirteenth to sixteenth embodimentsexcept for parts described below. In the following, differences from thesixteenth embodiment is mainly described.

First, contents of processes of the feature region extraction unit 290in the digital watermark detection apparatus 200G are described.

FIG. 94 is a figure for explaining processes of the feature regionextraction unit in the seventeenth embodiment of the present invention.

The feature region extraction unit 290 in this embodiment memorizes(stores in some way) a position and a shape of the feature regioncorresponding to a time when detection status is good in previousdigital watermark detection trial. When searching the currently receivedframe image for a feature region, the feature region extraction unit 290searches only a neighborhood of a previous feature region. Cameramovement is not so large in a capturing time interval for two images forwhich subtraction calculation is performed. Therefore, when previousdetection status is good, a correct feature region exists in aneighborhood of the previous feature region in a current frame image. Byusing this principle, more stable feature region extraction is realizedby narrowing the feature region search area.

By the way, “detection status is good” indicates a state in which adetection reliability indicator value in the document 1 is significantfor some extent but digital watermark detection does not yet succeed, ora state in which detection of a data block succeeds when performingdetection of each data block in the eighth embodiment. That is,“detection status is good” indicates a case in which existence ofdigital watermark is significantly obvious from detection status of thedigital watermark detection unit 240. In addition, a size of theneighborhood when performing the search may be changed according to thecapturing time interval between the two images for which subtraction iscalculated. For example, when the capturing time interval is short,since it can be predicted that movement amount of the camera is small,the size of the neighborhood is determined to be small. In contrast,when the capturing time interval is long, since it can be predicted thatthere may be a case in which the movement amount of the camera is large,the size of the neighborhood may be determined to be large.

Contents of processes of the detection subject region extraction unit220 are described.

FIG. 95 is a figure for explaining processes of the detection subjectregion extraction unit in the seventeenth embodiment of the presentinvention.

Also when extracting a detection subject region, in the same way as thefeature region extraction unit 290, the detection subject regionextraction unit memorizes (stores in some way) a position and a shape ofthe detection subject region corresponding to a time when detectionstatus is good in previous digital watermark detection trial. Whensearching the currently received feature region difference image for adetection subject region, the unit searches only a neighborhood of aprevious feature region. Accordingly, it can be also performed stably tosearch for the detection subject region.

By the way, when applying this embodiment to the sixteenth embodiment,since distortion factor due to camera movement and the like is absorbedby the feature region extraction process, a previous detection subjectregion is almost the same as a current detection subject region as shownin FIG. 95. Thus, the neighborhood size can be taken to be small. As apossible factor for failing extraction of the detection subject region,there is a case in which, when the moving image itself is a sceneincluding movement, not only the watermark pattern but also movementcomponent of moving images appears as noise in the difference. For thiscase, by using a small size neighborhood, search for the detectionsubject region becomes very stable.

When applying this embodiment to the thirteenth to fifteenthembodiments, since digital watermark detection is difficult when cameramovement is large in the first place, the neighborhood area can bedetermined to be small also in this case so that search for thedetection subject region becomes very stable.

In addition, it is also easy to apply this embodiment to the detectionsubject region extraction unit 220 in the digital watermark detectionapparatus in the first to twelfth embodiments. In this case, it is onlynecessary to, when detection status for a previous digital watermarkdetection trial is good, search the detection subject region in acurrent frame image starting from the neighborhood of the previousdetection subject region.

By the way, in this embodiment, although an example is shown for usingthe case “detection status was good in the previous digital watermarkdetection trial”, it is not limited to “previous”, and a feature regionor a detection region “at a temporally near time when good detectiontrial of digital watermark is performed” may be used.

When a state in which digital watermark detection information is notgood continues for a predetermined time, processes similar to those ofthe thirteenth to sixteenth embodiments may be performed withoutperforming the neighborhood search.

Effects of this Embodiment

In the present embodiment, in addition to effects of the thirteenth tofifteenth embodiments, since stability and reliability of the featureregion and the detection subject region improve, improvement of digitalwatermark detection performance is realized. Especially, when the methodfor detecting the feature region or the detection subject region is notso robust, remarkable effect is obtained so that digital watermarkdetection performance substantially improves.

Eighteenth Embodiment

This embodiment is the same as the fourteenth to seventeenth embodimentsexcept for parts described below. In the following, differences from thefourteenth embodiment are mainly described.

Processes of the detection subject region extraction unit 220 in thedigital watermark detection apparatus of the present embodiment aredescribed.

FIGS. 96-99 are drawings for explaining processes of the detectionsubject region extraction unit in the eighteenth embodiment of thepresent invention.

The process for extracting the detection subject region from thedifference frame image is performed from a difference image betweenframes of the watermark embedded moving images of the fourteenthembodiment. In this case, in the fourteenth embodiment, a frame patternin which the outside is bright and the inside is dark is added to thebasic watermark pattern in order to perform detection subject regionextraction accurately beforehand (by the way, as described in the firstembodiment, bright/dark is used for representing large/small of a pixelvalue, and it is not limited only to the brightness value). Whensearching for the detection subject region from the difference frameimage, the detection subject region extraction unit 220 checks pixelvalue change in the frame part. In the example of shown in FIG. 96,since the outside of the detection subject region is bright and theinside is dark, it is determined that the watermark pattern in thedetection subject region image is in phase (positive phase) with thebasic pattern when embedding, so that this phase information can be usedfor digital watermark detection process as is used in the fourthembodiment and the like. FIG. 97 shows an example in which the phase isreversed. By checking an edge part of the detection subject region, itcan be understood that positive phase/reversed phase can be determined.

In addition, examples in which the present embodiment is applied to thefifteenth embodiment are shown in FIGS. 98 and 99. In this embodiment,if a basic pattern of the positioning marker of the QR code (registeredtrademark) has a positive correlation with a positioning pattern foundfrom the detection subject region (FIG. 98), it is determined to be thepositive phase, and if they have a negative correlation, it isdetermined to be the reversed phase (FIG. 99). When it is reversedphase, digital watermark detection may be tried after reversing thephase of the detection subject region image. The reason for reversing isthat there are many cases that the existing two-dimensional code readingmethod does not support recognition of phase reversed code. Thus, phasereversing is necessary for using the existing two-dimensional codereading process.

For applying this embodiment to the sixteenth embodiment, it is obviousthat it is only necessary to perform processes similar toabove-mentioned processes except that the detection subject regionextraction unit 220 receives the feature region difference image insteadof the difference frame image.

Effects of the Present Embodiment

In the present embodiment, positive phase/reversed phase of thewatermark pattern in the detection subject region image is determinedbased on whether the phase of a pattern part of an extracted detectionsubject region is the same as that of the basic watermark pattern,wherein the detection subject region is extracted using the pattern suchas a frame or a positioning marker added to the basic watermark patternfor improving reliability for extracting the detection subject region.By using the phase information, detection performance can be improved asshown in the fourth embodiment and the like.

In addition, according to the present embodiment, the pattern such asthe frame and the positioning marker undertakes two roles for improvingreliability of extracting the detection subject region and for obtainingthe phase information, which means the pattern serves a dual purpose.That is, since it becomes unnecessary to shorten bit length of watermarkinformation by adding a flag to the watermark information for adjustingthe phase or to multiplex a phase determination signal that becomesnoise for the watermark information, digital watermark detectionperformance improves.

Nineteenth Embodiment

The present embodiment is the same as the thirteenth to eighteenthembodiments except for parts described below.

FIG. 100 shows a configuration of the digital watermark detectionapparatus in the nineteenth embodiment of the present invention.

The digital watermark detection apparatus 200H shown in the figureincludes an moving image input unit 210, a difference image generationunit 230, a detection subject region extraction unit 220, a digitalwatermark detection unit 240, a frame image buffer 301, and a differenceframe image buffer 303.

Differences from the thirteenth embodiment are processing contents ofthe detection subject region extraction unit 220 and that the differenceframe image buffer 303 is newly provided.

FIG. 101 is a flowchart showing operation of the digital watermarkdetection apparatus in the nineteenth embodiment of the presentinvention.

Step 1701) The analog or digital moving images are input by the movingimage input unit 210, so that a frame image is obtained sequentially.When inputting analog moving images, camera, scanner, or analog videosignals are input so as to obtain the frame image. When digital movingimages are input, the frame image is obtained by performing decodingprocesses.

Step 1702) Next, the difference image generation unit 230 calculates adifference between the currently obtained frame image and a previouslyobtained frame image stored in the frame image buffer 301 so as togenerate a difference frame image.

Step 1703) The currently input frame image is buffered into thedifference frame buffer 301.

Step 1704) The currently obtained difference frame image is added andstored into the difference frame buffer 301.

Step 1705) A detection subject region that is a subject for watermarkdetection is extracted from the difference frame image that is added andstored in the difference frame buffer 301 so as to obtain a detectionsubject region image.

At this time, as shown in FIG. 102, the difference frame image may beadded and stored in the difference frame image buffer 303 after changingpixel values of the difference frame image into absolute values. Then,the detection subject region is extracted from the difference frameimage added and stored in the difference frame image buffer 303, anddistortion correction and size normalization may be performed on theimage of the detection subject region of the current difference frameimage to obtain the detection subject region image. According to thismethod, since the absolute value image of the difference frame image isadded and stored, contrast between the detection subject region and abackground area increases by the adding and storing irrespective of thephase of the watermark pattern in the received difference frame image sothat reliability for extracting the detection subject region increases.

Alternatively, in a case where the difference frame image can begenerated in synchronization with the switching timing of the watermarkpattern represented by the watermark pattern switching information likethe third embodiment, it cannot be determined whether the watermarkpattern in the obtained difference frame image is in phase/reversedphase with respect to the basic watermark pattern using the start timingof capturing and the like, but it can be determined whether thecurrently obtained difference frame image is in phase with a previouslyobtained difference frame image (for example, in the example of FIG. 31,in-phase status can be obtained every other difference image). By usingthis, as shown in FIG. 103, the difference image may be added and storedin the difference frame image buffer 303 after aligning the phase of thedifference image (as a result, only images in phase with the basicwatermark pattern are added or only images having reversed phase areadded). Then, the detection subject region is extracted after changingthe difference frame image added and stored in the difference frameimage buffer into absolute values, and distortion correction and sizenormalization may be performed on the image of the detection subjectregion of the difference frame image added and stored in the differenceframe image buffer 303 to obtain a detection subject region image.According to this method, the watermark pattern is emphasized by theadding and storing, an effect that digital watermark detectionperformance improves can be obtained in addition to the effect that thereliability for extracting detection subject region improves.

Step 1706) Next, the digital watermark detection unit 240 tries todetect digital watermark from the detection subject region image tooutput a detection result.

Step 1707) Digital watermark detection does not succeed, the movingimage input unit obtains a next frame image to repeat theabove-mentioned processes sequentially.

In addition, the digital watermark detection apparatus where the presentembodiment is applied to the sixteenth embodiment is described.

FIG. 104 shows a configuration of the digital watermark detectionapparatus in the nineteenth embodiment of the present invention.

The digital watermark detection apparatus 200I shown in the figureincludes an moving image input unit 210, a feature region extractionunit 290, a difference image generation unit 230, a detection subjectregion extraction unit 220, a digital watermark detection unit 240, afeature region buffer 302, and a feature region difference image buffer304.

Differences from the sixteenth embodiment are processing contents of thedetection subject region extraction unit 220 and that the feature regiondifference image buffer 304 is newly provided.

FIG. 105 is a flowchart (applied to the sixteenth embodiment) showingoperation of the digital watermark detection apparatus in the nineteenthembodiment of the present invention.

Step 1801) When the moving image input unit 210 receives analog movingimages displayed on a TV and the like and video-captured in real time bya camera, or digital moving images encoded by MPEG, the moving imageinput unit 210 obtains a frame image sequentially. When inputting analogmoving images, camera, scanner, or analog video signals are input so asto obtain the frame image. When digital moving images are input, theframe image is obtained by performing decoding processes.

Step 1802) Next, the feature region extraction unit 290 extracts afeature region in the frame image. For extracting the feature region, atechnique such as rectangular area detection by edge recognition and thelike shown in the document 3 can be used. Distortion caused by camerataking angle and the like is corrected for the extracted feature region,and the size is normalized so that the image is output as a featureregion extraction image.

Step 1803) Next, the difference image generation unit 230 calculates adifference between the currently obtained feature region image and apreviously obtained feature region image stored in the feature regionimage buffer 302 so as to generate a feature region difference image.

Step 1804) The current feature region image is buffered into the featureregion image buffer 302 in preparation for a next detection trial.

Step 1805) The detection subject region extraction unit 220 stores thefeature region difference image into the feature region difference imagebuffer 304.

At this time, as shown in FIGS. 106 and 107, the feature regiondifference image is added and stored in the feature region differenceimage buffer 304 after aligning the phase of the feature regiondifference images. For aligning the phase, the above-mentioned methodfor synchronizing with the watermark pattern switching timing and themethods shown in the sixth and seventeenth embodiments can be used. As aresult, only images in phase with the basic watermark pattern are addedor only images having reversed phase are added in the feature regiondifference image buffer 304.

Step 1806) Next, a detection subject region that is a subject forwatermark detection is extracted from the feature region differenceimage added and stored in the feature region difference image buffer304, and the detection subject region image is obtained after performingdistortion correction and size normalization.

Step 1807) Next, the digital watermark detection unit 240 tries todetect digital watermark from the detection subject region image tooutput a detection result.

Step 1808) When digital watermark detection does not succeed, the movingimage input unit 210 obtains a next frame image to repeat theabove-mentioned processes sequentially.

As mentioned above, according to the method for adding and storing afteraligning the phase, since the watermark pattern in the detection subjectregion relatively stands out compared with a case using only onedifference image, reliability of extracting the detection subject regionimproves and the detection performance of the digital watermark improveslike the sixth embodiment.

Effects of the Present Embodiment

In the present embodiment, the detection subject region is extracted byrelatively strengthening the watermark pattern by storing the differenceframe image or the feature region difference image so that improvementof reliability for extracting the detection subject region is realized.

In addition, especially, by performing adding and storing after aligningthe phase of the watermark pattern in the difference image, not onlyreliability of the detection subject can be improved but also detectionperformance of digital watermark can be improved simultaneously.Generally, since the watermark pattern is added to an image with a weakamplitude such that a human cannot perceive the watermark pattern, thereis a possibility that the detection subject region is not clearlyidentified in one difference image. According to the present invention,this problem can be solved by the above-mentioned contrivances.

Twentieth Embodiment

The present embodiment is the same as the first to nineteenthembodiments except for parts described below. In the following,differences from the thirteenth embodiment are mainly described.

Processes of the watermark pattern superimposing unit 130 in the digitalwatermark embedding apparatus 100 in the present embodiment aredescribed.

FIGS. 108 and 109 are figures to explain the processes of the watermarkpattern superimposing unit in the twentieth embodiment of the presentinvention.

The watermark pattern superimposing unit 130 receives a frame image andwatermark patterns generated based on the frame display time. It isassumed that a plurality of watermark patterns are generated such that aplurality of watermark patterns can be superimposed on one frame imageas necessary. For example, assuming that three watermark patterns eachhaving different information are provided. The watermark patternsuperimposing unit 130 adjusts an amplitude of each watermark pattern asnecessary, and superimposes each watermark pattern onto the frame imageafter adjusting the position and the size of the watermark pattern foran object, in the frame image, desired to be associated with informationindicated by the watermark pattern. FIG. 108 shows an example forsuperimposing watermark patterns onto three different object regions inthe frame image. Alternatively, as shown in FIG. 109, a watermarkpattern corresponding to an object may be superimposed onto a watermarkpattern corresponding to the whole image like nesting. In this case, thevalues of the inside watermark pattern in the nest may be overwritten onthe values of the outside watermark pattern. In addition, as shown inFIG. 109, like the fourteenth and fifteenth embodiments, a marker andthe like for extracting the detection subject region is added to thewatermark pattern.

Next, processes of the difference image generation unit 230 of thedigital watermark detection apparatus in the present embodiment aredescribed.

FIG. 110 is a figure for explaining processes of the difference imagegeneration unit according to the twentieth embodiment of the presentinvention.

The process itself of the difference image generation unit 230 iscompletely the same as that of the thirteenth embodiment. But, sincethere are a plurality of watermark patter regions in the received frameimage, a plurality of watermark pattern regions are obtained in thedifference frame image as shown in FIG. 110.

Next, processes of the detection subject region extraction unit 220 inthe present embodiment are described.

FIGS. 111 and 112 are figures for explaining the processes of thedetection subject region extraction unit in the twentieth embodiment ofthe present invention. Like the thirteenth embodiment, by searching forrectangular areas after calculating absolute values of the pixel valuesof the difference frame image, three detection subject regions can befound as shown in FIG. 111. A detection subject region image isgenerated for each of the three detection subject regions, and digitalwatermark detection trial is performed for each detection subject regionimage.

Processes of the detection subject region extraction unit 220 whenperforming watermark pattern superimposition as shown in FIG. 109 aredescribed using FIG. 112. Even though watermark pattern regions arenested in the difference frame image, the detection subject regions ofthe nesting structure can be found by using a positioning marker and thelike shown in the fourteenth of fifteenth embodiment, so that thedetection subject region image is generated for each detection subjectregion to perform digital watermark detection trial. By the way, in FIG.112, the inside watermark pattern is nested in the outside watermarkpattern so that any pattern of the outside watermark pattern does notremain in the inside region. But, digital watermark detection isavailable since loss of the pattern can be supported because ofrobustness of the digital watermark scheme itself.

Next, an example of feedback output of the digital watermark detectionapparatus 200 in the present embodiment is shown.

FIGS. 113-116 show examples of the feedback output of the digitalwatermark detection apparatus in the twentieth embodiment of the presentinvention.

The detection status obtained for each detection subject region issynthesized on a preview screen in accordance with the position and sizeof the detection subject region on the display screen of the digitalwatermark detection apparatus 200. In FIG. 113, digital watermarkdetection status detected from three detection subject regions aresuperimposed and displayed at object positions associated with originalwatermark patterns.

FIG. 114 shows an example of synthesizing of the digital watermarkdetection statuses detected from nested two detection subject regions.When a plurality of digital watermark detection subject regions areobtained like this, it is possible to feedback the digital watermarkdetection statuses at the same time.

In addition, as shown in FIGS. 115 and 116, when zooming in a part ofthe display screen to take the part, a digital watermark detectionstatus corresponding to only the part can be output. By using this, forexample, an instruction can be realized for selecting some objects froma plurality of objects in the display screen using digital watermarkdetection.

By the way, it is obvious that the present embodiment can be alsocarried out by arbitrarily combining with contrivances of the first tonineteenth embodiments.

Effects of the Present Embodiment

According to the present embodiment, a plurality of watermark patternsare superimposed and embedded onto one frame of the moving images, and aplurality of detection subject regions in the frame image obtained bycamera-capturing are extracted so that digital watermark detection isperformed for each region. Accordingly, it becomes possible to embed anddetect digital watermark associated with an object in the frame image sothat convenience improves. In addition, by performing capturing anddetection by zooming in a particular object when taking the image, itcan be used for an user interface for selecting a particular object inthe screen.

Twenty First Embodiment

In the present embodiment, an example is described for performingembedding of watermark information as phase difference change of thewatermark pattern.

The present embodiment is the same as the first to twentieth embodimentsexcept for parts described below. In the following, differences from thefirst embodiment are mainly described.

The watermark pattern generation unit 120 in the digital watermarkembedding apparatus 100 in the present embodiment is described.

FIG. 117 is a figure for explaining an example of watermark patternswitching information in the twenty first embodiment of the presentinvention.

The watermark pattern switching information in the present embodiment isnot information for directly instructing to reverse the pattern, but isinformation for indicating a period of the watermark pattern and aswitching point in the period.

FIGS. 118 and 119 are figures for explaining processes of the watermarkpattern switching unit in the twenty first embodiment of the presentinvention.

The watermark pattern switching unit 122 first converts gray values ofthe received basic watermark pattern into a phase difference changepattern. For example, when the basic watermark pattern is binary asshown in FIG. 118, two phase difference change values corresponding tobright and dark respectively are obtained, the phase difference changevalue is associated to a pixel position having a pixel value of brightor dark on the basic watermark pattern. As to how the binary phasechange difference values are obtained, when the watermark patternswitching information indicates “switching intermittently every 1/10second such that 3/10 second becomes one period”, since one period is3/10 second, and the switching timing is 1/10 second, if phase ischanged by 2π/3 in 1/10 second, the phase returns to the original phaseafter 3/10 second. Therefore, two values of 2π/3 and −2π/3 which is inthe opposite direction are used as the phase difference change values.

After obtaining the phase difference change pattern as mentioned above,the watermark pattern switching unit 122 adds the phase differencechange pattern to a pattern of a next previous watermark phase for eachswitching timing where display time of received frame is placed on atime axis of the pattern switching information. In the example of FIG.118, a phase difference change pattern is added to the watermark phasepattern at each of times t0, t1 and t2. By the way, it is assumed thatan initial value of the watermark pattern in a head frame of movingimages, for example, is set such that each element of the watermarkphase pattern arbitrarily takes any of 0 and the above-mentioneddifference change values.

After obtaining the watermark phase pattern in the above-mentioned way,the watermark pattern switching unit 122 converts the watermark phasepattern into a watermark pattern. As to how the conversion is performed,as shown in FIG. 119 for example, each element value of the watermarkphase pattern is associated with a point on a circle having a radius r(r is a given value) centered on the origin point (0, 0) in a coordinatesystem having orthogonal axes indicating more than one components ofimage, that are Cb and Cr in YCbCr color coordinate system in theexample of the figure. A Cb coordinate and a Cr coordinate in the Cb-Crcoordinate system obtained as a result of the above-mentioned processare regarded as a pixel value of the watermark pattern corresponding toan element of the watermark phase pattern so as to obtain all pixelvalues of the watermark pattern. The watermark pattern obtained in theabove mentioned way becomes a pattern in which each pixel has aplurality of component values.

By the way, in the present embodiment, although Cb and Cr are used as anexample, various methods may be used such as R-G of RGB color coordinatesystem, X-Y in XYZ color coordinate system, H-S in HSV color coordinatesystem and the like.

When visual sensitivity for change amount is different for each axis,scales may be corrected as necessary such that visual sensitivitybecomes the same for each change amount for each axis. In the presentembodiment, an example using Cb and Cr components is shown because theycannot be easily perceived compared with Y in the YCbCr color coordinatesystem.

In addition, when the present embodiment is carried out using a singlecomponent instead of a plurality of image components, for example, usinga brightness component, digital watermark embedding using phasedifference change can be realized in the same way as the presentembodiment by using the waveform pattern for the basic watermark patternlike the modulation method (A-2) of the first embodiment and by changingthe phase of the waveform pattern.

Similar to the first embodiment, the watermark pattern obtained in theabove-mentioned way is sequentially superimposed onto the frame image sothat the watermark embedded moving images can be obtained.

The difference image generation unit 230 in the digital watermarkdetection apparatus 200 in this embodiment is described.

FIG. 120 is a figure for explaining processes of the difference imagegeneration unit in the twenty first embodiment of the present invention.The difference image generation unit 230 obtains the difference image byperforming processes similar to those of the first embodiment. Afterthat, the difference image generation unit 230 further modifies thedifference image. As an example of the method of the modification, asshown in FIG. 120, Cb and Cr component values (cb,cr) is obtained foreach pixel of the difference image, and the values are plotted on aCb-Cr orthogonal coordinate system to obtain polar coordinaterepresentation (R, θ) in the same way as performed when embedding. Next,an angle among angles (π/6, π/2, π/6, 5π/6, 7π/6, 3π/2, 11π/6),indicated by circles in the figure, nearest to θ is determined so that asign (+ or −) assigned to the nearest angle is obtained. Finally, adistance R from the origin point is multiplied by this sign to obtain avalue, and the value is overwritten on the difference image as a pixelvalue of the pixel. This process is performed on every pixel of thedifference image.

Processes after the difference image generation unit 230 are the same asthose in the first embodiment.

Next, principle of the present embodiment is described.

As assumption of explanation, it is assumed that capture timing anddifference timing are specified based on watermark pattern switchinginformation like the third embodiment as shown in FIG. 121 in thedigital watermark detection apparatus in the present embodiment. In FIG.121, it is specified to calculate a difference between detection subjectregion images 1/10 apart. As described in the fourth embodiment and thelike, in the digital watermark embedding and detection method describedin the first to twentieth embodiments, the watermark pattern in theobtained difference image becomes in phase or in reversed phase withrespect to the phase used when embedding so that it is necessary tosupport it by using various contrivances.

Principle of the present embodiment is described with reference to FIGS.122 and 123. FIG. 122 enumerates all possible patterns of Cb-Crcomponent values embedded in two frame 1/10 second apart, namely, valuescorresponding to phase pattern used when embedding and Cb-Cr valuesobtained by subtraction, when the phase difference change pattern valueused when embedding is 2π/3. As shown in the figure, when the phasedifference change value when embedding is 2π/3, the Cb-Cr componentvalue in the difference image has a phase of one of three phases: π/6,5π/6 and 3π/2.

FIG. 123 shows a case when the phase difference change value is −2π/3.In this case, the Cb-Cr component value in the difference image haseither one phase of three phases: 7π/6, π/2 and 11π/6. That is, Cb-Crcomponent values in the difference image can be classified to two groupsaccording to the phase difference change values. By using this, thephase difference change value when embedding can be steadily obtainedirrespective of timing of capturing by determining which group the phaseof the Cb-Cr component when performing detection belongs to as shown inFIG. 120. Since the phase difference change value corresponds to a valueof a term in a series obtained by spreading the watermark information,the watermark pattern (corresponding to the watermark pattern in thecorrected difference image in FIG. 120) obtained by subtraction is notreversed irrespective of capturing timing so that the watermark patternis in phase with the basic watermark pattern used when embedding.

The characteristic that “the watermark pattern obtained by subtractionis always constant irrespective of timing” is extremely important. Forexample, as to the modulation method (A-1) in the first embodiment,since bits of detected watermark information are reversed when thewatermark pattern in the difference image is reversed, contrivances suchas those in the fourth and fifth embodiments are required foridentifying the phase.

In addition, as to other modulation methods in the first embodiment, thephase needs to be adjusted for improving detection performance of thewatermark information and the detection subject region so that variouscontrivances are used as shown in the sixth embodiment and thenineteenth embodiment for example. In contrast, by using the presentembodiment, such problem is completely solved, and since the watermarkpattern is constant irrespective of how the difference is calculated,bit reversal does not occur in the detected watermark information as amatter of course, and it is only necessary simply add images withoutconcern of the phase when adding and storing the difference images.

In addition, when watermark information is divided into data blocks sothat they are sequentially embedded in a time direction like the eighthembodiment, there is a risk in that incorrect watermark information maybe detected for a difference image that straddles a border of the datablocks as shown in FIG. 124 according to the modulation method shown inthe first embodiment. Therefore, for keeping safety, it is necessary toput a section where watermark is not embedded into data block sections.Accordingly, there is a problem in that the watermark information lengthper a unit time is reduced.

However, by using the present embodiment, watermark information isrepresented by the phase difference change. Thus, by performingembedding by adding a phase difference change representing current datablock information to a phase pattern at a next previous data blockterminal end, even though a difference image straddling a data blockborder is obtained when performing detection, data block information canbe correctly detected from the difference image. Accordingly, sinceembedding/detection can be performed while switching data blockscontinuously without inserting unnecessary blanks, watermark informationamount per a unit time can be increased.

By the way, it is obvious that the present embodiment can be carried outby arbitrarily combining it with each contrivance of the first totwentieth embodiments.

Effects of the Present Embodiment

According to the present embodiment, since watermark information isembedded as the phase difference change, the difference image includingconstant watermark pattern can be obtained irrespective of the timingfor obtaining the difference image. Accordingly, it becomes unnecessaryto avoid bit reversal of watermark information and to adjust phases whenstoring difference images so that it is only necessary to simply add andstore difference images. Thus, apparatus configuration can besimplified, speed of processes can be increased, and the phasesynchronization signal becomes unnecessary, so that increase ofwatermark information length and improvement of tolerance can berealized. In addition, when using the data block using type like theeighth embodiment, reliability of detection information can be keptwithout inserting any unnecessary blank between data blocks. Thus, thewatermark information length per a unit time can be increased.

Twenty Second Embodiment

The present embodiment is the same as the first to twenty firstembodiments except for parts described below.

In the following, differences from the first embodiment is described.Processes of the difference image generation unit 230 in the digitalwatermark detection apparatus 200 are described.

FIG. 125 is a figure for explaining processes of the difference imagegeneration unit in the twenty second embodiment of the presentembodiment.

In the present embodiment, after the difference image generation unit230 generates the difference image of the detection subject regionimages, the difference image generation unit 230 performs filteringprocess on the difference image to output it.

FIGS. 126A and 126B are figures for explaining examples of filteringprocess in the twenty second embodiment of the present invention. Thedifference image is represented as one dimensional signal forexplanation. In FIG. 126A, non-linear filtering process is performed inwhich, when an absolute value of a pixel value of the difference imageis equal to or greater than a given threshold, the value is clipped tothe threshold value. In the FIG. 126B, non-linear filtering process isperformed in which, when an absolute value of a pixel value of thedifference image is equal to or greater than a given threshold, thepixel value is changed to 0.

Effects of the Present Embodiment

According to the present embodiment, large difference component thatoccurs when taking moving images including movement can be suppressed bythe filtering process. Generally, since the watermark pattern isembedded with a weak amplitude, a difference component caused bymovement of the image itself appears as a difference pixel value havinga large absolute value on the difference image. Therefore, by using thenon-linear filter in the present embodiment, the difference componentdue to the movement of the moving images can be suppressed, and thecomponent of the watermark pattern is not suppressed. Thus, an S/N ratioof the watermark pattern is improved so that detection performance fromthe moving images including movement can be improved.

Twenty Third Embodiment

The present embodiment is the same as the first to twenty secondembodiments except for parts described below. In the following,differences from the first embodiment are described.

FIG. 127 is a figure for explaining processes of the watermark patternsuperimposing unit in the twenty third embodiment of the presentinvention.

The watermark superimposing unit 130 in the present embodiment includesa function for storing a several number of past frame images that aresequentially received, and generates a difference image between acurrently received frame image and a previous frame image. The previousframe image may be a next previous one or may be one several framesbefore. Next, the watermark pattern is scaled as necessary like thefirst to twenty second embodiments. Next, an amplitude of the watermarkpattern is adjusted based on the difference image. As an only processafter that, the watermark pattern in which the amplitude is adjusted issuperimposed on the frame image like the first to twenty secondembodiments.

FIG. 128 is a figure for explaining amplitude adjustment for thewatermark pattern in the twenty third embodiment of the presentinvention. In (1) in the figure, a total sum of absolute values of pixelvalues of the difference image are obtained, and the amplitude of thewhole watermark pattern is adjusted based on the value of the total sum.As to how the adjustment is performed, the greater the total sum valueis, the greater the amplitude is adjusted to be, such as amplifying theamplitude by a value in proportion to the total sum value, for example.Alternatively, as shown in (2) in the figure, there may be a method inwhich, based on a pixel value of each coordinate point in the differenceimage, the amplitude of a pixel value at a corresponding coordinatepoint in the watermark pattern may be adjusted. In this case, forexample, the larger the absolute value of the pixel value of thedifference image is, the larger the amplitude of the watermark patternat a corresponding position is adjusted to be. In both of (1) and (2) inthe figure, the larger the number of pixels having a large absolutevalue is in the difference image, that is, the more vigorously themovement in the moving image is, the more the amplitude of the watermarkpattern is amplified. In addition, the method for amplifying may beperformed in more complicated way such as performing (2) in the figureafter performing (1) in the figure.

Effects of the Present Embodiment

According to the present embodiment, when superimposing a watermarkpattern onto moving images including movement, a watermark patternhaving larger amplitude is superimposed for a frame including manypixels having a large absolute value in the difference value, that is,for a frame including vigorous movement compared with a frame includingsmall movement. Generally, noise is hardly perceived in a moving imagescene including vigorous movement compared with a still scene. Thus,even when the amplitude of the watermark pattern is strengthened in theframe including the vigorous movement, it is possible that image qualitydeterioration due to watermark embedding cannot be perceived.

In addition, in a scene including especially many movements, since thereare many pixels having a large absolute value in the difference imagewhen performing detection, there is a problem in that noise becomeslarge as to watermark detection. But, by embedding watermark patternstrongly beforehand for a scene including vigorous movement like thepresent embodiment, the S/N ratio improves so that watermark detectionperformance improves.

Twenty Fourth Embodiment

The present embodiment is the same as the first to twenty thirdembodiments except for parts described below. In the following,differences from the first embodiment are described.

FIG. 129 shows a configuration of a digital watermark detectionapparatus in the twenty fourth embodiment of the present invention.

The digital watermark detection apparatus 200J shown in the figureincludes a moving image input unit 210, a detection subject regionextraction unit 220, a difference image generation unit 230, a digitalwatermark detection unit 240, a detection subject region image buffer250, and a zooming process unit 310. The configuration shown in thefigure is one in which the zooming process unit 310 is added to thedigital watermark detection apparatus 100A of the first embodiment.

FIG. 130 is a flowchart of operation in the twenty fourth embodiment ofthe present invention.

Operation in steps 1901-1906 in the flowchart shown in the figure is thesame as that in steps 201-206 in FIG. 20, and the explanation is notprovided.

Step 1907) When detection status of the digital watermark detection unit210 is not good, the process goes to step 1901, and when it is good,process in step 1908 is performed.

Step 1908) When the detection status is good, the zooming process unit310 generates a zooming parameter such that the detection subject regionbecomes a given size using the detection subject region informationindicating the size and the position of the detection subject region andthe detection status of the digital watermark detection unit 240, andthe zooming parameter is given to the moving image input unit 210, andthe process goes to the process of step 1901. Accordingly, the movingimage input unit 210 automatically causes a camera, a scanner and thelike to perform zooming based on the given zooming parameter.

Here, in the same way as the seventeenth embodiment, “detection statusis good” indicates a status in which the detection reliability indexvalue of the document 1 is a significant value for some extent butdigital watermark detection is not succeeded, for example, or indicatesa status in which detection of a data block is succeeded when performingdetection for each data block in eighth embodiment, that is, itindicates a case in which existence of digital watermark issignificantly obvious according to a detection status of the digitalwatermark detection unit 240. For example, when the detection subjectregion represented by current detection subject region information isless than 40%, zooming-in is automatically performed such that itbecomes equal to or greater than 40% such that the area of the detectionsubject region becomes equal to or greater than 40% of the area of theframe image. Alternatively, when detection status in a detection trialis good and when the area of the detection subject region is 20%,zooming-in may be gradually performed for each detection trial so as tochange display image smoothly to improve user interface.

In a case when detection of the edge of the detection subject regionfails if performing zooming-in too much, the detection subject regionmay not be zoomed in until it fills the frame image but it may be zoomedin to a given size (50% of the frame image, for example). Further,instead of to the given size, the zooming ratio may be changed asnecessary such that vertexes or sides of the detection subject regiondefined by current detection subject region information are fallenwithin the frame image (for example, between a case when the detectionsubject region exists in a center part of the frame image and a casewhen it exists in an edge side, degree of zooming is smaller in thelatter case).

By the way, in the above-mentioned example, although differences fromthe first embodiment are described, a combination with any of thethirteenth to fifteenth embodiments can be easily realized, and it canbe easily realized to perform zooming such that the feature region ofthe sixteenth embodiment becomes a given size, to combine theseventeenth embodiment by zooming into a range of neighborhood searchaccording to change of zooming ratio in the same way, or to detectwatermark information associated with a particular image region byperforming detection trial by framing one of a plurality of watermarkpatterns into a center part of the taken image by automaticallyperforming the zooming in the twentieth embodiment.

Effects of the Present Embodiment

According to the present embodiment, when a pixel size of the detectionsubject region in the taken frame image is small, a detection subjectregion of a larger pixel size can be obtained by automaticallyperforming zooming-in. Generally, when a number of pixel samples becomessmall, digital watermark detection becomes difficult. Thus, byincreasing the number of pixel samples in the detection subject regionby performing zooming-in, performance for detecting digital watermarkcan be improved.

In addition, by performing zooming-in only when the detection status isgood based on the detection status of the digital watermark, forexample, when detection status is good as shown in FIG. 131A, the numberof pixel samples in the detection subject region can be increased byzooming-in so as to be able to increase detection performance. At thesame time, as shown in FIG. 131B, when a region that is not a realdetection subject region is incorrectly determined as a detectionsubject region, since digital watermark detection status is not good,zooming-in is not performed, that is, since zooming-in is performed onlyfor a detection subject region where existence of digital watermark issurly obvious to improve detection performance, unnecessary zooming-inis not performed so that usability can be improved.

Twenty Fifth Embodiment

In the present embodiment, watermark information is embedded as thephase difference change. When detecting, a difference image (A) isgenerated first, and a phase difference between temporally adjacentdifference images (A) is measured, and digital watermark detection isperformed from a difference image (B) obtained by using the phasedifference, so that digital watermark detection is always availableunder a same condition irrespective of capture timing, capture timeinterval or shatter speed when performing detection.

The present embodiment is the same as the first to twenty fourthembodiments except for parts described below. In the following,differences from the first embodiment are mainly described. Theconfiguration of the digital watermark embedding apparatus in thepresent embodiment is the same as one in the first embodiment.

FIG. 132 is a figure for explaining an example of watermark patternswitching information in the twenty fifth embodiment of the presentinvention.

Watermark pattern switching information of the watermark patterngeneration unit 120 in the digital watermark embedding apparatus 100 inthe present embodiment is not information for directly instructingreversal of pattern, but is represented by a continuous function such asa sine wave in which one cycle is 6/30 second, more particularly, isinformation indicating a period of a sine wave. Actually, the inputmoving images are provided as discrete frame images having a specificframe rate such as 30 frames/second. Thus, as shown in a lower figure inFIG. 132, embedding can be performed by performing discrete sampling ona sine wave. By the way, in the present embodiment, a phase value for aperiod of the sine wave is used as shown in the following description.

FIG. 133 is a figure for explaining processes of the watermark patternswitching unit in the twenty fifth embodiment. The watermark patternswitching unit 122 converts density values of the input basic watermarkpattern into a sign pattern first. The sign pattern is an arrayincluding each element value that is obtained by extracting + or − signof each density value of the basic watermark pattern and by changing thesize to 1.

Next, the watermark pattern switching unit 122 obtains a phase (to becalled a watermark pattern switching phase hereinafter) for a period ofwatermark pattern switching information corresponding to a frame displaytime, where input frame display time is taken on a time axis of thewatermark pattern switching information. Then, a phase change valuebetween the watermark pattern switching phase and a watermark patternswitching phase of a next previous frame is obtained (+π/3 in FIG. 133),and this phase change value is multiplied by the element value of thesign pattern to obtain a phase change amount of each element value ofthe watermark phase pattern. The watermark phase pattern is an arrayhaving a size the same as that of the sign pattern, wherein each elementhas a phase value from 0 to 2π. The initial value of each element of thewatermark phase pattern may be an arbitrary value.

After obtaining the watermark phase pattern as mentioned above, thewatermark pattern switching unit 122 converts the phase pattern into awatermark pattern. As to how conversion is performed, as shown in FIG.134 for example, each element value of the watermark phase pattern isassociated with a point on a circle of a radius r (r is a given value)centered on the origin (0, 0) in a coordinate system having orthogonalaxes of more than one image components, that are, Cb and Cr in the YCbCrcolor coordinate system in the figure, wherein the angle of the point isthe phase pattern element value. All pixel values of the watermarkpattern are obtained in which a Cb coordinate and a Cr coordinate on theCb-Cr coordinate system obtained as a result of the above-mentionedprocess are a pixel value of the watermark pattern corresponding to anelement of the phase pattern. The watermark pattern obtained in such away becomes a pattern including a plurality of component values for eachpixel. By the way, although Cb and Cr are used as an example in thepresent embodiment, various methods can be used such as using R-G in theRGB color coordinate system, using X-Y in the XYZ color coordinatesystem, and using Hue-Saturation in the HSV color coordinate system (inthis case, Hue-Saturation may not be taken on the orthogonal coordinate,but, a phase value may directly set to be a Hue value and may have agiven Saturation value).

In addition, when visual sensitivity for change amount is different foreach axis, scales may be corrected as necessary such that visualsensitivity becomes the same for each change amount for each axis (forexample, an ellipse instead of a circle is used in FIG. 134). In thepresent embodiment, an example using Cb and Cr components is shownbecause they cannot be easily perceived compared with Y in the YCbCrcolor coordinate system.

In addition, when the present embodiment is carried out using a singlecomponent instead of a plurality of image component, for example, usinga brightness component, digital watermark embedding using phasedifference change can be realized in the same way as the presentembodiment by using the waveform pattern for the basic watermark patternand changing the phase of the waveform pattern like the modulationmethod (A-2) of the first embodiment.

Similar to the first embodiment, the watermark pattern obtained in theabove-mentioned way is sequentially superimposed onto the frame image sothat the watermark embedded moving images can be obtained.

When Cb-Cr is used, to adjust the amplitude of the watermark patternmeans to increase or decrease the amplitude (distance between point a0and the origin in FIG. 134) on the Cb-Cr coordinate system. In addition,when using Hue-Saturation, it means to increase or decrease Saturation.In addition, when multiple values instead of binary value are used forthe basic watermark pattern, the amplitude of a pixel value of thewatermark pattern may be increased or decreased according to a size ofan absolute value of the corresponding pixel value of the basicwatermark pattern.

FIG. 135 is a block diagram of the digital watermark detection apparatusin the twenty fifth embodiment of the present invention.

The digital watermark detection apparatus in the present embodimentincludes an moving image input unit 210, a detection subject regionextraction unit 220, a difference image generation unit 230, a phasedifference measurement unit 360, a digital watermark detection unit 240,a detection subject region image buffer 250, and a phase pattern buffer370.

FIG. 136 is a flowchart of operation of the digital watermark detectionapparatus in the twenty fifth embodiment of the present invention.

Differences from the first embodiment are described. A difference image(represented as difference image (A) in the present embodiment) outputby the difference image generation unit 230 in the same way as the firstembodiment is input to the phase difference measurement unit 360 asshown in FIG. 137. The phase difference measurement unit 360 generates aphase pattern from the difference image (A) (step 2040), measures aphase difference between the currently obtained phase pattern and apreviously obtained phase pattern so as to form a new difference image(B) based on the phase difference and output it (step 2050). The digitalwatermark detection unit 240 receives the difference image (B) andperforms digital watermark detection from the difference image (B) inthe same way as the first embodiment (step 2070).

FIG. 138 is a figure for explaining contents of processes of the phasedifference measurement unit in the twenty fifth embodiment of thepresent invention.

The phase difference measurement unit 360 receives the difference image(A). The phase difference measurement unit 360 divides the differenceimage (A) into blocks in the same way performed when embedding, andobtains each total sum for the Cb component and the Cr component foreach block, and obtains an amplitude R and an phase θ when representingthe result (Cb, Cr) as a point on the Cb-Cr coordinate system, and setseach of these to be a value of an element corresponding to the blockposition in a phase pattern. The phase pattern is an array including anamplitude and a phase for each element position, and each element isassociated with each block in the difference image.

Next, the phase difference measurement unit 360 compares the previouslyobtained phase pattern stored in the phase pattern buffer 370 and thecurrently obtained phase pattern so as to determine a sign that is plusor minus based on a phase difference of each element, further determinea new amplitude using a monotone increasing function in which the largereach amplitude of the two phase pattern is, the larger value thefunction takes, and determines each element value of the differenceimage (B) having the sign and the amplitude as the element value.

FIGS. 140A and B are figures for explaining contents of processing ofthe phase difference measurement unit in the twenty fifth embodiment ofthe present invention.

A phase difference θ−θ′ between a phase value θ of an element of thecurrently obtained phase pattern and a phase value θ′ of a correspondingelement of the previously obtained phase pattern is obtained, and if thevalue is within a range of 0−π, a sign “+” is selected, and if the valueis within a range of −π−0, a sign “−” is selected. As an example of themonotone increasing function used for obtaining a new amplitude value ρfrom an amplitude value R of an element of the currently obtained phasepattern and an amplitude value R′ of a corresponding element of thepreviously obtained phase pattern, ρ(R,R′)=R+R′ or ρ(R,R′)=R×R′ can beused. The reason for using monotone increasing function with respect tothe amplitude values R and R′ is to improve detection performance ofdigital watermark by assigning weight to one in which watermark signalis strongly remained, that is, amplitude R, R′ is large.

After performing the processes for selecting the sign and obtaining theamplitude value for every element, the difference image (B) is sent tothe digital watermark detection unit to try watermark detection from thedifference image (B). By the way, the currently obtained phase patternis stored in the phase pattern buffer 370 for using in later detectiontrial.

Next, principle of the present embodiment is described using someexamples where capturing timing is different when performing detection.

In the capturing example A in FIG. 141, frame capturing is performedevery 1/15 second, that is, every other frame. Considering a Cb-Cr valuein a watermark pattern in a difference image (A) (represented as (a),(b), (c), . . . in FIG. 141) obtained between adjacent captured frames(represented as (1), (2), (3), (4) . . . , in FIG. 141), the valuebecomes a result of calculating a difference when regarding Cb-Crcomponent value as a vector as shown in FIG. 142 as to a positiveelement in the sign pattern in FIG. 133, and as shown in FIG. 143 as toa minus element. This is because, calculation for the difference imagecorresponds to difference calculation of each term of the Cb componentand Cr component (accordingly, original image component is largelyreduced), and as described in the twenty first embodiment, a pluralityof phases are obtained in the difference image (A) as shown in FIGS. 142and 143.

In the present embodiment, after obtaining the difference image (A), thephase pattern is generated on the basis of the phase value of the Cb-Crcomponent of the difference image (A), and a difference value betweentemporally adjacent phase patterns is measured (phase difference databetween the phase pattern is represented as (α), (β), (γ) . . . in FIG.141). The statuses are shown in FIG. 144 (when the sign pattern is plus)and FIG. 145 (when the sign pattern is minus). In FIGS. 144 and 145, byobtaining the difference of the phases, it turns out that a phasedifference 2π/3 of the watermark pattern when embedding is equallyobtained between any adjacent captured frames from the phase differenceof the phase patterns.

In this embodiment, digital watermark detection is performed bydetermining the sign of the difference image (B) using plus and minus ofthe phase difference value. As mentioned above, since the phasedifference is associated with the sign pattern when embedding, it isobvious that detection of digital watermark is available. That is,digital watermark detection can be similarly performed from any timingof the captured frames.

FIG. 146 shows a capturing example B different from the capturingexample A (FIG. 141). In the capturing example B, although capturing isperformed every 1/15 second like the capturing example A, timing ofstart of capturing is shifted. In the case of the capturing example B,when the difference image (A) is generated first, since phase of thewatermark pattern embedded in each captures frame is shifted withrespect to the capturing example A (FIG. 141), combination of Cb-Crvalues (phase values of phase pattern) in the watermark pattern in thedifference image (A) is different from the cases (FIG. 142 and FIG. 143)of the capturing example A. Accordingly, in the twenty first embodimentin which watermark information is detected only from combination ofCb-Cr values (phase values of the phase pattern) in the watermarkpattern in the difference image (A), it turns out that, the sign pattercan be classified to “+” or “−”, but which is “+” or which is “−” amongthe two groups can not be identified.

However, in the present embodiment, by measuring the phase difference ofadjacent difference images (A) as shown in FIGS. 149 and 150, a sign thesame as the sign pattern element when embedding can be always detected.That is, according to the present embodiment, there is no differencebetween the capturing example A and the capturing example B, so thatdigital watermark detection can be performed similarly in an arbitrarycapture start timing.

FIG. 151 shows a capturing example C different from each of thecapturing examples A and B. In the capturing example C, the timing forcapturing is not constant in which the timing is 1/30 second interval or1/15 second interval or the like. In the case of capturing example C,when the difference image (A) is generated first, Cb-Cr values (phasevalues in the phase pattern) in the difference image (A) are obtainedwhere the combination is different from those of the capturing example Aand capturing example B as shown in FIGS. 152 and 153. In the case wheretime intervals for capturing are different for each time like thecapturing example C, since it is difficult to classify Cb-Cr values(phase values in the phase pattern) in the difference image (A) intotwo, detection is difficult using a method like that in the twenty firstembodiment.

However, in the present embodiment, the phase difference betweenadjacent difference images (A) is measured as shown in FIGS. 154 and155. At this time, although the value of the phase difference variesaccording to change of the capturing time interval, the sign determinedbased on the phase difference is constant (0˜π→+, −π˜0→−) irrespectiveof the capturing time interval, and the sign is the same as the signpattern when embedding. Accordingly, even when the capturing timeintervals change like the capturing example C, digital watermarkdetection can be performed without problem.

FIG. 156 shows a capturing example D different from each of thecapturing examples A, B and C. In a case such as camera capturing formoving images, when shatter speed is low, there is a case in which aplurality of moving image frames are exposed and superimposed on onecapture frame. For example, in the capturing example D shown in FIG.156, moving image frames displayed every 1/30 second are superimposed onone capture frame by multiple exposure. However, since the phase of thewatermark pattern in the captured frame obtained by the superimpositionis a superimposition of phases in each of the multiplexed moving imageframes (since synthesized image can be obtained by calculating weightingaverage with a proper ratio for each term of the Cb and Cr components ina plurality of frames), a value in the middle of the phase of each ofthe multiplexed moving image frames is taken as shown in FIG. 157(although FIG. 157 shows an example in which two frames are synthesizedwith a ratio of 1:1 by multiple exposure, it is obvious that, in a casewhen 2:1 for example, an intermediate value is taken according to aninterior division ratio). In the case of the capturing example D, whenthe difference image (A) is obtained first, Cb-Cr values (phase valuesin the phase pattern) are obtained in which the combination is differentfrom that of each of the capturing examples A, B and C as shown in FIGS.158 and 159. In the case of the capture frame obtained by multipleexposure like the capturing example D, it is difficult to classify Cb-Crvalues (phase values in the phase pattern) in the difference image (A)into two, detection is difficult using a method like that in the twentyfirst embodiment.

However, according to the present embodiment, by measuring the phasedifference between adjacent difference images (A) as shown in FIGS. 160and 161, a sign the same as the sign pattern element when embedding canbe always detected. That is, according to the present embodiment, evenif a plurality of moving image frames are multiple-exposed andsynthesized in a captured frame when capturing timing shifts by anarbitrary time smaller than frame display timing of the moving images,digital watermark detection can be performed similarly.

The characteristic that “the watermark pattern obtained by thedifference is always constant irrespective of capture timing, capturetime interval or shatter speed” is extremely important. For example, asto the modulation method (A-1) in the first embodiment, when thewatermark pattern in the difference image is reversed, since bits of thedetection watermark information are reversed, contrivance like thefourth or fifth embodiment are required for identifying phase.

In addition, as to other modulation methods shown in the firstembodiment, various contrivances are used as shown in the sixth or ninthembodiment, for example, since it is necessary to adjust phases whenstoring difference images in order to improve detection performance forwatermark information and the detection subject region. But, by usingthe present embodiment, such problems are completely solved, and sincethe watermark pattern is constant irrespective of how the difference iscalculated, bit reversal does not occur in the detected watermarkinformation as a matter of course, and it is only necessary to simplyadd images without concern of the phase when adding and storing thedifference images.

In addition, when watermark information is divided into data blocks sothat it is sequentially embedded in a time direction like the eighthembodiment, there is a risk in that incorrect watermark information maybe detected for a difference image that straddles a border of the datablocks as shown in FIG. 124 in the modulation method shown in the firstembodiment. Therefore, for keeping safety, it is necessary to put asection where watermark is not embedded into data block sections.Accordingly, there is a problem in that the watermark information lengthper a unit time is reduced.

However, by using the present embodiment, watermark information isrepresented by the phase difference change. Thus, by performingembedding by adding a phase difference change representing current datablock information to a phase pattern at a next previous data blockterminal end, even though a difference image straddling a data blockborder is obtained when performing detection, data block information canbe correctly detected from the difference image. Accordingly, sinceembedding/detection can be performed while switching data blockscontinuously without inserting unnecessary blanks, watermark informationamount per a unit time can be increased.

In addition, in a case where capture interval is changed and a pluralityof moving image frames are synthesized to be a capture frame by multipleexposure, which are difficult case for the twenty first embodiment,according to the present embodiment, detection is available withoutproblem as mentioned above, and effects by combining with each of theabove-mentioned embodiments can be obtained.

By the way, it is obvious that the present embodiment can be carried outby combining it with each contrivance in the first to twenty fourthembodiments. That is, before-mentioned effects can be obtained byinserting the process of the phase difference measurement unit of thepresent embodiment into a corresponding portion in each embodiment (suchas before the digital watermark detection unit or the detection subjectregion extraction unit) and by performing the process. When beingcombined with other embodiment, it is only necessary to read thedifference image (B) in the present embodiment as the difference imageor the detection subject region image. In the following, a simpleexplanation is provided for a case where contrivance of the presentembodiment is applied to each embodiment.

First to twelfth, and twenty first to twenty fourth embodiments: outputof the difference image generation unit in the digital watermarkdetection unit is supplied to the phase difference measurement unit asthe difference image (A) to obtain the difference image (B) and it issupplied to the digital watermark detection unit.

Thirteenth to twentieth embodiments: a detection subject region imagethat is an output of the detection subject region extraction unit issupplied to the phase difference measurement unit as the differenceimage (A) of the present embodiment to obtain the difference image (B)and it is supplied to the digital watermark detection unit.

Effects of the Twenty Fifth Embodiment

According to the present embodiment, watermark information is embeddedas the phase difference change, and the difference image (A) isgenerated first when detecting, and the phase difference betweentemporally adjacent difference images (A) is measured, and digitalwatermark detection is performed from the difference image (B) obtainedby using the phase difference. Therefore, as mentioned before, digitalwatermark detection can be always performed under a same conditionirrespective of capture timing, capture time interval, or shatter speedwhen detecting. Accordingly, it becomes unnecessary to avoid bitreversal of watermark information and to adjust phases when storingdifference images so that it is only necessary to simply add and storedifference images. Thus, apparatus configuration can be simplified,speed of processes can be increased, and the phase synchronizationsignal becomes unnecessary, so that increase of watermark informationlength and improvement of tolerance can be realized.

In addition, when using the data block using type like the eighthembodiment, reliability of detection information can be kept withoutinserting any unnecessary blank between data blocks. Thus, the watermarkinformation length of per a unit time can be increased. In addition,when capture timing of a detection apparatus changes or when a pluralityof moving image frames are multiple-exposed and synthesized by camerataking, digital watermark detection is available without any problem.Thus, the configuration of the moving image input unit 210 in thedigital watermark detection apparatus 200 can be simplified (it is notnecessary to perform capturing with accurate clocks), and it becomespossible to improve digital watermark detection performance in a casewhen using analog optical reading such as a case using a camera.

Twenty Sixth Embodiment

According to the present embodiment, watermark information is embeddedas the phase difference change, and the difference image (A) isgenerated first when detecting, and the phase difference betweentemporally adjacent difference images (A) is measured, a differenceimage (B) obtained by using the phase difference is generated, adetection subject region is extracted by adding and storing thedifference images (B), and digital watermark detection is performed fromthe detection subject region image. Therefore, as mentioned before,digital watermark detection can be always performed under a samecondition irrespective of capture timing, capture time interval, orshatter speed when detecting. In addiction, effects of distortion causedby camera taking angle is corrected and watermark tolerance is improvedby storing, so that more stable digital watermark detection is realized.

FIG. 162 is a block diagram of the digital watermark embedding apparatusin the twenty sixth embodiment of the present invention.

The digital watermark embedding apparatus 100 includes a frame imageobtaining unit 110, a watermark pattern generation unit 120, a watermarkpattern superimposing unit 130, and a moving image data reconstructionunit 140. The digital watermark embedding apparatus 100 of the presentembodiment receives original moving image data, watermark informationand watermark pattern switching information.

FIG. 163 is a flowchart of operation of the digital watermark embeddingapparatus of the twenty sixth embodiment of the present invention.

Step 100) As shown in FIG. 164, the frame image obtaining unit 110sequentially obtains a frame image and the frame display time from theoriginal image data one by one. The frame display time, for example, maybe one indicating absolute time, from the head of the moving image, thatis determined from time code and frame rate when reproducing, or may beone by which relative time interval between frames when reproducing canbe measured. When the supplied moving image data is coded data such asMPEG data, the frame image obtaining unit 110 obtains the frame imageafter performing decoding and the like.

Step 110) Next, the watermark pattern generation unit 120 generates thewatermark pattern using the watermark information, frame display timeand watermark pattern switching information.

Step 120) Next, the watermark pattern superimposing unit 130superimposes the watermark patter onto the frame image to generate awatermark embedded frame image.

Step 130) Finally, the moving image data reconstruction unit 140reconstructs a series of the watermark embedded frame images that aresequentially generated as moving image data so as to output the seriesas watermark embedded moving image data. At this time, encoding such asMPEG encoding may be performed as necessary.

FIG. 165 shows a configuration of the watermark pattern generation unitin the twenty sixth embodiment of the present invention. FIG. 166 is aflowchart of operation of the watermark pattern generation unit in thetwenty sixth embodiment of the present invention.

The watermark pattern generation unit 120 includes a basic watermarkpattern generation unit 121 and a watermark pattern switching unit 122,and receives the watermark information, the watermark pattern switchinginformation and the frame display time.

Step 2100) First, the basic watermark pattern generation unit 121converts the watermark information into a basic watermark pattern thatis a two-dimensional pattern. There may be various methods for theconversion as shown in the first embodiment, and the modulation shown inthe first embodiment is used in the present embodiment for simplifyingexplanation.

FIG. 167 shows an example of the process of the basic watermark patterngeneration unit 121. In the present embodiment, a method is used inwhich a size of a pixel value is associated with a value of seriesobtained by directly performing spread spectrum modulation on the bitvalue of the watermark information using spreading series. Each pixelvalue of the basic watermark pattern in this case takes a value that isplus or minus.

Step 2110) Next, the watermark pattern switching unit 122 in thewatermark patter generation unit 120 determines necessity of phasereversal of the basic watermark pattern based on relationship betweenthe frame display time and the watermark pattern switching information,and changes the phase of the basic watermark pattern as necessary andoutput it as a watermark pattern.

FIG. 168 is a block diagram of the watermark pattern switching unit inthe twenty sixth embodiment of the present invention.

The watermark pattern switching unit 122 includes a sign patterngeneration unit 121, a watermark pattern switching phase change valuecalculation unit 1222, a watermark phase difference pattern generationunit 1223, a watermark phase pattern generation unit 1224 and awatermark phase pattern imaging unit 1225.

FIG. 169 is a flowchart of the operation of the watermark patternswitching unit in the twenty sixth embodiment of the present invention.

Step 2200) The sign pattern generation unit 1221 in the watermarkpattern switching unit 122 converts the basic watermark pattern into asign pattern first.

Step 2210) Next, the watermark pattern switching phase change valuecalculation unit 1222 obtains a difference value of the watermarkpattern switching phase represented by the watermark pattern switchinginformation based on a time difference between a next previous frame andthe current frame represented by frame display times, in which thedifference value is regarded as the watermark pattern switching phasechange value.

Step 2220) Next, the watermark phase difference pattern generation unit1223 generates a watermark phase difference pattern that has the signindicated by the sign pattern and has the watermark pattern switchingphase change value as the absolute value.

Step 2230) Next, the watermark phase pattern generation unit 1224obtains a current watermark phase pattern by providing the currentwatermark phase difference pattern as a phase difference to each elementvalue of the watermark phase pattern for the next previous frame.

Step 2240) Finally, the watermark phase pattern imaging unit 1225converts the watermark phase pattern into an image pattern and outputsthe result as a watermark pattern.

FIG. 170 is a figure for explaining watermark pattern switchinginformation in the twenty sixth embodiment of the present invention.

Watermark pattern switching information of the watermark patterngeneration unit 120 in the digital watermark embedding apparatus 100 inthe present embodiment is not information for directly instructingreversal of pattern, but is represented by a continuous function such asa sine wave in which one cycle is 6/30 second, more particularly, isinformation indicating a period of a sine wave. Actually, the inputmoving images are provided as discrete frame images having a specificframe rate such as 30 frames/second. Thus, as shown in a lower figure inFIG. 170, embedding can be performed by performing discrete sampling ona sine wave. By the way, in the present embodiment, a phase value for aperiod of the sine wave is used as shown in the following description.

FIG. 171 is a figure for explaining processes of the watermark patternswitching unit in the twenty sixth embodiment of the present invention.

The watermark pattern switching unit 122 converts density values of theinput basic watermark pattern into a sign pattern first. The signpattern is an array including each element value that is obtained byextracting + or − sign of each density value of the basic watermarkpattern and changing the size to 1.

Next, the watermark pattern switching unit 122 obtains a phase (to becalled a watermark pattern switching phase hereinafter) for a period ofwatermark pattern switching information corresponding to a frame displaytime where input frame display time is taken on a time axis of thewatermark pattern switching information. Then, a phase change valuebetween the watermark pattern switching phase and a watermark patternswitching phase for a next previous frame is obtained (+π/3 in FIG. 171)as the watermark pattern switching phase change value, and thiswatermark pattern switching phase change value is multiplied by theelement value of the sign pattern to obtain a watermark phase differencepattern. Then, each element value of the watermark phase pattern isprovided to each element value of a watermark phase pattern for a nextprevious frame as a phase change amount so that a current watermarkphase pattern is obtained. The watermark phase pattern is an arrayhaving a size the same as that of the sign pattern, wherein each elementhas a phase value from 0 to 2π). The initial value of each element ofthe watermark phase pattern may be an arbitrary value.

After obtaining the watermark phase pattern as mentioned above, thewatermark phase pattern imaging unit 125 converts the watermark phasepattern into a watermark pattern. As to how conversion is performed, asshown in FIG. 172 for example, each element value of the watermark phasepattern is associated with a point on a circle of a radius r (r is agiven value) centered on the origin (0, 0) in a coordinate system havingorthogonal axes of more than one image components, that are, Cb and Crin the YCbCr color coordinate system in the figure, wherein the angle ofthe point is the phase pattern element value. All pixel values of thewatermark pattern are obtained in which a Cb coordinate and a Crcoordinate on the Cb-Cr coordinate system obtained as a result of theabove-mentioned process are a pixel value of the watermark patterncorresponding to an element of the phase pattern. The watermark patternobtained in such a way becomes a pattern including a plurality ofcomponent values for each pixel.

By the way, although Cb and Cr are used as an example in the presentembodiment, various methods can be used such as using R-G in the RGBcolor coordinate system, using X-Y in the XYZ color coordinate system,and using Hue-Saturation in the HSV color coordinate system (in thiscase, Hue-Saturation may not be taken on the orthogonal coordinate, but,a phase value may directly set to be a Hue value and may have a givenSaturation value).

When visual sensitivity for change amount is different for each axis,scales may be corrected as necessary such that visual sensitivitybecomes the same for each change amount for each axis (for example, anellipse instead of a circle is used in FIG. 134). In the presentembodiment, an example using Cb and Cr components is shown because theycannot be easily perceived compared with Y in the YCbCr color coordinatesystem. In addition, when the present embodiment is carried out using asingle component instead of a plurality of image component, for example,using a brightness component, digital watermark embedding using phasedifference change can be realized in the same way as the presentembodiment by using the waveform pattern for the basic watermark patternand changing the phase of the waveform pattern like the modulationmethod (A-2) of the first embodiment.

Similar to the first embodiment, as shown in FIG. 173, the watermarkpattern obtained in the above-mentioned way is sequentially superimposedonto the frame image while changing the amplitude of the watermarkpattern as necessary in the watermark pattern superimposing unit 130 sothat a watermark embedded moving image can be obtained. At this time,the size of the watermark pattern is changed to a size equal to or lessthan that of the frame image so that the watermark pattern issuperimposed on a center of the frame image. By the way, when Cb-Cr isused, to adjust the amplitude of the watermark pattern means to increaseor decrease the amplitude (distance between point a0 and the origin inFIG. 172) on the Cb-Cr coordinate system. In addition, when usingHue-Saturation, it means to increase or decrease Saturation.

In addition, when multiple value instead of binary value is used for thebasic watermark pattern, the amplitude of a pixel value of the watermarkpattern may be increased or decreased according to a size of an absolutevalue of the corresponding pixel value of the basic watermark pattern.Finally, the moving image data reconstruction unit 140 reconstructs thesequentially generated watermark embedded frame images as moving imagedata to output watermark embedded moving image data. When performingreconstruction, encoding such as MPEG encoding may be performed.

Above descriptions are descriptions for the digital watermark embeddingapparatus.

Next, the digital watermark detection apparatus in the presentembodiment is described.

FIG. 174 is a block diagram of the digital watermark detection apparatusin the twenty sixth embodiment of the present invention.

The digital watermark detection apparatus includes a moving image inputunit 210, a feature region extraction unit 290, a difference imagegeneration unit 230, a phase difference calculation unit 360, adetection subject region extraction unit 220, a digital watermarkdetection unit 240, a feature region image buffer 302, a phase patternbuffer 370 and a difference image (B) storing buffer 390.

The digital watermark detection apparatus 200 receives analog movingimages displayed on a TV and the like or digital moving images encodedby MPEG.

FIG. 175 is a flowchart of operation of the digital watermark detectionapparatus in the twenty sixth embodiment of the present invention.

Step 2300) The moving image input unit 210 receives analog or digitalmoving images to obtain a frame image sequentially. For inputting theanalog moving images, camera signals, scanner signals or analog videosignals are received to obtain the frame image. For digital movingimages, the frame image is obtained by performing decoding process andthe like.

Step 2310) Next, the feature region extraction unit 290 extracts afeature region in the captured frame image to obtain a feature regionimage in the same way as the sixteenth embodiment and the like.

Step 2320) Next, the difference image generation unit 230 obtains adifference image between the currently obtained feature region image anda previously obtained feature region image stored in the feature regionimage buffer 302 to output as it a difference image (A).

Step 2330) In addition, in preparation for next detection trial, thecurrent feature region image is buffered in the feature region imagebuffer 302.

Step 2340) Next, the phase difference measurement unit 360 converts thedifference image (A) into a phase pattern.

Step 2350) The phase difference measurement unit 360 measures a phasedifference between the currently obtained phase pattern and a previouslyobtained phase pattern stored in the phase pattern buffer 370, and newlygenerates a difference image (B) based on the phase difference andamplitude values of the two phase patterns.

Steps 2360-2380) Next, the detection subject region extraction unit 220adds and stores the current difference image (B) in the difference image(B) storing buffer 390, and extracts a detection subject region from thestored difference image (B) to obtain a detection subject region image.

Step 2390) Finally, the digital watermark detection unit 240 tries todetect digital watermark from the detection subject region image tooutput a detection result.

Step 2400) When digital watermark detection does not succeed, the movingimage input unit 210 obtains a next frame image to repeat theabove-mentioned processes sequentially.

FIG. 176 is a figure for explaining processes of the feature regionextraction unit in the twenty sixth embodiment of the present invention.

The feature region extraction unit 290 receives the frame image andextracts a feature region in the frame image in the same way as thesixteenth embodiment to obtain a feature region image.

FIG. 177 is a figure for explaining processes of the difference imagegeneration unit in the twenty sixth embodiment of the present invention.The difference image generation unit 230 receives a feature regionimage. The difference image generation unit 230 generates a differenceimage between the currently received feature region image and apreviously obtained feature region image stored in the feature regionimage buffer 302 to output it as the difference image (A). Thesubtraction process is performed by subtraction for each component suchas R, G, B or Y, Cb, Cr. In addition, in preparation for a nextdetection trial, the current feature region image is buffered in thefeature region image buffer 302.

FIG. 178 is a figure for explaining process contents of the phasedifference calculation unit in the twenty sixth embodiment of thepresent invention.

The phase difference measurement unit 360 receives the difference image(A). The phase difference measurement unit 360 obtains a Cb componentand a Cr component for each pixel of the difference image (A), andobtains an amplitude R and a phase θ by representing the result (Cb, Cr)as a point on the Cb-Cr coordinate system, and sets these to be valuesof an element corresponding to the pixel position of the phase pattern.The phase pattern is an array including an amplitude and a phase foreach element position, and each element is associated with each pixel inthe difference image.

Next, as shown in FIG. 179, the phase difference measurement unit 360compares the previously obtained phase pattern stored in the phasepattern buffer 370 and the currently obtained phase pattern so as todetermine a sign that is plus or minus based on a phase difference ofeach element, further determines a new amplitude using a monotoneincreasing function in which the larger each amplitude of the two phasepatterns is, the larger value the function takes, and determines eachelement value of the difference image (B) having the sign and theamplitude as its value.

FIGS. 180A and B show examples of methods for selecting the sign basedon the phase difference. A phase difference θ−θ′ between a phase value θof an element of the currently obtained phase pattern and a phase valueθ′ of a corresponding element of the previously obtained phase patternis obtained, and if the value is within a range of 0−π, a sign “+” isselected, and if the value is within a range of −π−0, a sign “−” isselected. As an example of the monotone increasing function used forobtaining a new amplitude value ρ from an amplitude value R of anelement of the currently obtained phase pattern and an amplitude valueR′ of a corresponding element of the previously obtained phase pattern,ρ(R,R′)=R+R′ or ρ(R,R′)=R×R′ can be used. The reason for using amonotone increasing function with respect to the amplitude values R andR′ is to improve detection performance of digital watermark by assigningweight to one in which watermark signal is strongly remained, that is,amplitude R, R′ is large. After performing the processes for selectingthe sign and obtaining the new amplitude value for every element, thedifference image (B) is output. In addition, in preparation for a nextdetection trial, the currently obtained phase pattern is stored in thephase pattern buffer 370. Next, contents of processes of the detectionsubject region extraction unit 220 are described.

The detection subject region extraction unit 220 receives the differenceimage (B). The detection subject region extraction unit 220 adds andstores the received difference image (B) in the difference image (B)storing buffer 390.

FIG. 181 is a figure for explaining the difference image (B) storingbuffer in the twenty sixth embodiment of the present invention. Thedifference image (B) storing buffer 390 is a pixel array having a sizethe same as that of the difference image (B). Each pixel value of anewly received difference image (B) is added to a corresponding pixelposition in the difference image (B) storing buffer 390.

FIG. 182 is a figure for explaining processes of the detection subjectregion extraction unit in the twenty sixth embodiment of the presentinvention. After the difference image (B) is added and stored in thedifference image (B) storing buffer 390, the detection subject regionextraction unit 220 converts pixel values in the difference image (B)storing buffer 390 to absolute values, and extracts the detectionsubject region from the difference image (B) storing buffer 390 in thesame way as the thirteenth embodiment and the like. Then, distortion ofthe image of the detection subject region in the difference image (B)storing buffer 390 before being converted into absolute values iscorrected such that it becomes a given size rectangle, and further thesize is normalized to generate and output the detection subject regionimage.

Effects of the detection subject region extraction unit 220 obtained bythe above-mentioned processes are described. Considering the differenceimage (B) obtained by the phase difference measurement unit 360,absolute values of pixel values in a pixel region where the watermarkpattern is embedded are generally larger than those in a pixel regionwhere the watermark pattern is not embedded. The reason is as follows.Absolute values of pixel values in a pixel region where the watermarkpattern is embedded in the difference image (A) are large since thewatermark pattern is different for each captured frame, but absolutevalues of pixel values in a pixel region where the watermark pattern isnot embedded are small since temporal correlation of original imagecomponents is high. In addition, since the phase difference measurementunit 360 determines a sign of each pixel value of the difference image(B) using the phase difference, a same sign is always obtained in a samepixel position in the pixel region where the watermark pattern isembedded irrespective of capturing timing as described in the twentyfifth embodiment. Therefore, by adding and storing images in thedifference image (B) storing buffer 390, the absolute value becomeslarger every time the addition is performed. However, as to the pixelregion where the watermark pattern is not embedded, since signs do notbecome the same and vary and absolute values of the pixel values aresmall as mentioned before, absolute values of the pixel values becomenot so large by the addition and storing so that contrast between thepixel area where the watermark pattern is not embedded and the pixelarea where the watermark pattern is embedded becomes relatively largerevery time the addition and storing are performed. Thus, by convertingthe pixel values obtained by the addition and storing in the differenceimage (B) buffer 390 into absolute values, difference between plus andminus of the watermark pattern disappears so that the pixel region wherethe watermark pattern is embedded emerges with high contrast in thepixel region that is a background region where the watermark pattern isnot embedded. Accordingly, the region where the watermark pattern isembedded can be extracted as the detection subject region reliably.

In addition, since the watermark pattern is relatively strengthened bythe addition and storing as shown in FIG. 181, detection performance indigital watermark detection described next can be improved, and digitalwatermark detection having higher tolerance can be realized.

Finally, the digital watermark detection unit 240 receives the detectionsubject region image, and tries to detect digital watermark from thedetection subject region image. A process example of the digitalwatermark detection unit 240 is described using FIG. 183. FIG. 183 showsa demodulation method when detecting corresponding to the modulationmethod (A-1) when embedding digital watermark (document 1).

First, the detection subject region image is divided into blocks thenumber of which is the same as that of blocks of the watermark patternwhen embedding. Next, sum of pixel values in each block is obtained, andthe sums are arranged in an order of the blocks to obtain a detectionsubject series that is a one-dimensional series. Next, a sectioncorresponding to 1 bit when embedding is extracted from the detectionsubject series, and correlation calculation between the section and acorresponding section in a spreading series is performed. When thecorrelation value is a large plus value, it is determined that a bitvalue “1” is detected, and when the correlation value is a large minusvalue, it is determined that a bit value “0” is detected. Accordingly,the detection process is performed for every bit. In addition, errorcorrection/detection decoding process may be performed on watermarkinformation for which detection process is completed as necessary. Whendigital watermark detection does not succeed, the process returns to theframe capturing process of the moving image input unit 210 next so as torepeat the detection trial. Or, when input of the moving images ends,detection process is terminated.

Effects of Twenty Sixth Embodiment

According to the present embodiment, effects that there is no problemeven if the moving images are captured at any timing can be obtained inthe same way as the twenty fifth embodiment. In addition to that,effects the same as those of the nineteenth embodiment can be alsoobtained by obtaining the difference image (B) from the difference image(A) for each pixel, adding and storing the difference image (B) andextracting the detection subject region from the difference image (B)that is converted to absolute values to perform digital watermarkdetection.

In addition, since the watermark information is embedded as the phasedifference change, pixel values can be added as they are in the additionand storing of the difference image (B), and since bits of the watermarkinformation do not reversed, simpler and more efficient digitalwatermark scheme is obtained compared with the nineteenth embodiment.

In addition, according to the present embodiment, a part, in thecaptured frame image, from which digital watermark detection should beperformed is identified using various contrivances. Thus, digitalwatermark detection can be available reliably and under variousconditions of image taking angles and background images, and digitalwatermark detection performance is further improved. Accordingly, itbecomes possible to perform digital watermark detection process reliablyin real time from moving images on a TV and the like that are taken by acamera.

It is possible to construct each operation of the digital watermarkembedding apparatus 100 and the digital watermark detection apparatus200 as a program so as to install and execute the program in a computerused as the digital watermark embedding apparatus or the digitalwatermark detection apparatus, or, the program can be distributed via anetwork. In addition, the program can be stored in a computer readablemedium such as a CD-ROM and an electrical memory and the like and can beused.

The present invention is not limited to the specifically disclosedembodiments, and variations and modifications may be made withoutdeparting from the scope of the invention. The present invention can beapplied to digital watermark techniques. The present invention is notlimited only to the above-described embodiments, and the presentinvention can be applied to various fields such as a three-dimensionaldisplay system, a three-dimensional picture-taking system, voiceinformation and the like, and a signal region such as infrared rays thatcannot be perceived.

[Effects]

According to an embodiment of the present invention, by embedding thewatermark pattern that temporally changes with respect to frames ofmoving images, a pattern that switches at high speed instead of a fixedpattern is embedded in each frame in the moving images. Thus, since itis known that visual sensitivity is low for high frequency in timefrequency characteristics in human visual characteristics, image qualitydeterioration due to watermark embedding can be decreased.

In addition, when performing detection, digital watermark detection isperformed by generating the difference image between temporally adjacenttwo captured frames for frame images continuously received from ananalog input device such as a camera. Thus, the original image signalthat becomes noise for the digital watermark signal can be largelycancelled and digital watermark detection performance can be improved.

In addition, if the watermark pattern switching timing used whenembedding is known in a detection side, each difference image wherewatermark pattern is remained can be selected using times of capturedtwo frames, so that it becomes unnecessary to perform unnecessarydigital watermark detection. In addition, by controlling capture timingof camera in a detection side or controlling time difference between twoimages when obtaining the difference image in accordance with thewatermark pattern switching timing, it becomes possible that thewatermark pattern always remains in the difference image so thatefficient digital watermark detection becomes possible.

In addition, by determining a phase of the watermark pattern in thedifference image, correct watermark information can be detected evenwhen using a modulation method of a type in which bit of detectionwatermark information is reversed due to reversal of the phase of thewatermark pattern. In the case, 1 bit in the watermark information maybe used as a flag for phase determination, or digital watermark forphase determination different from the watermark information may beused.

In addition, difference information sequentially obtained by continuouscapturing is added and stored after adjusting phases so as to relativelyincrease strength of the watermark pattern, then, digital watermarkdetection is tried from the storing buffer. Accordingly, detectionbecomes possible even when adequate detection performance is notobtained only from one difference image. By the way, for adjusting thephases, the above-mentioned flag for phase reversal determination may beused, the watermark for phase determination different from the watermarkinformation may be used, and in addition to that, there may be a methodusing sign polarity of the correlation value in spread spectrummodulation.

In addition, by adding and storing correlation values calculated whendetecting digital watermark and by evaluating stored correlation valuesso as to determine detection availability, detection can be performedeven when adequate detection performance from one difference imagecannot be obtained. In this case, the correlation values may be storedwith a constant phase by aligning phases of the difference images orabsolute values of the correlation values may be stored.

In addition, the embedding information is divided into small data blocksto perform embedding while switching the data block in a time direction,and data blocks obtained by performing detection for each data block areconnected when detecting. Thus, it becomes possible to embed/detectlonger watermark information. Further, in the same way as processes forone difference image, difference phase determination may be performedfor each data block detection, phase determination may be also performedby embedding the data block ID representing order of data blocks,difference images in which phases are aligned may be stored, orcorrelation values may be stored, so that it becomes also possible tofurther improve digital watermark detection performance.

Further, under a situation where real time detection processing isperformed using a camera, the status of the digital watermark detectionprocess is fed back and output in real time. Thus, convenience improvessince interactivity increases. Especially, it can be expected thatdetection performance improves by urging the user to make a status inwhich detection can be easily performed.

Further, by embedding watermark information as the phase differencechange of the watermark pattern, a difference image can be obtained suchthat it includes a constant watermark pattern irrespective of obtainingtiming of the difference image. Accordingly, bit reversal of watermarkinformation is prevented, and it becomes unnecessary to adjust phaseswhen storing the difference image or when storing the correlation value,and it is only necessary to simply add and store. Therefore, sinceapparatus configuration is simplified, speed of processing is increasedand the phase synchronization signal is unnecessary, watermarkinformation length can be increased and tolerance can be improved. Inaddition, as to the data block using type, since reliability ofdetection information is ensured even though unnecessary blank is notinserted between data blocks, length of watermark information per a unittime can be increased.

In addition, according to an embodiment of the present invention, whenembedding watermark pattern temporally changing with respect to framesof moving images, a watermark pattern smaller than a frame size of themoving image data is superimposed. In addition, when performingdetection, a difference image between captured frame images is generatedso as to extract a region having large difference, that is, a regionthat is a detection subject for digital watermark from the differenceimage. Accordingly, stable digital watermark detection can be realizedirrespective of picture taking angles or background images.

In addition, by extracting the detection subject region from adifference image between feature region images each obtained byperforming distortion correction and size normalization of a featureregion for each captured frame, the detection subject region can beextracted stably even through the camera moves while taking the images.

In addition, when embedding, a plurality of watermark patternsindicating different pieces of information are associated with objectsin the image, and are arranged and superimposed, and when performingdetection, a plurality of detection subject regions are extracted fromthe difference image, so that a plurality of pieces of digital watermarkinformation can be detected at the same time or different informationfor each object can be extracted.

In addition, watermark information is embedded as phase differencechange. When performing detection, a difference image (A) is generatedfirst, a phase difference between temporally adjacent difference images(A) is measured, and digital watermark detection is performed from adifference image (B) obtained using the phase difference. Therefore,digital watermark detection can be performed always under a samecondition irrespective of capture timing and capture time interval orshatter speed when performing detection.

In addition, watermark information is embedded as phase differencechange. When performing detection, a difference image (A) is generatedfirst, a phase difference between temporally adjacent difference images(A) is measured, a difference image (B) obtained by using the phasedifference is generated, a detection subject region is extracted byadding and storing the difference image (B), and digital watermarkdetection is performed from the detection subject region image.Therefore, as mentioned before, digital watermark detection can beperformed always under a same condition irrespective of capture timingand capture time interval or shatter speed when performing detection,and in addition to that, effects of distortion caused by picture takingangle are corrected and watermark tolerance is improved by storing, sothat it becomes possible to realize more stable digital watermarkdetection.

In addition, when watermark pattern switching timing used when embeddingis known in the detection side, a difference image in which a watermarkpattern remains can be selected using times of captured two frames, sothat it becomes unnecessary to perform useless digital watermarkdetection. Further, by controlling capture timing of a camera in thedetection side or controlling time difference of two images whenobtaining the difference image in accordance with watermark patternswitching timing, a watermark pattern always remains in the differenceimage, so that it becomes possible to perform efficient digitalwatermark detection.

In addition, according to the embodiment of the present invention, whenembedding a watermark pattern temporally changing with respect to framesof moving images, a watermark pattern smaller than a frame size of themoving image data is superimposed. In addition, when performingdetection, a difference image between captured frame images is generatedso as to extract a region having large difference, that is, a regionthat is a detection subject for digital watermark from the differenceimage, so that stable digital watermark detection can be realizedirrespective of picture taking angles or background images.

In addition, by extracting the detection subject region from adifference image between feature region images each obtained byextracting a feature region, performing distortion correction and sizenormalization for each captured frame, the detection subject region canbe extracted stably even through the camera moves while taking theimages.

In addition, when searching for the detection subject region or thefeature region, by searching only a neighborhood of a position wheredigital watermark detection status was good before, it becomes possibleto perform region extraction more stably.

In addition, by adding and storing difference images so as to extractthe detection subject region from the added and stored difference image,it becomes possible to perform further stable region extraction and toimprove digital watermark detection performance.

In addition, when embedding, a plurality of watermark patternsindicating different pieces of information are associated with objectsin the image, and the watermark patterns are arranged and superimposed,and when performing detection, a plurality of detection subject regionsare extracted from the difference image, so that a plurality of piecesof digital watermark information can be detected at the same time anddifference information for each object can be extracted.

In addition, when the pixel size in the detection subject region in theframe image when is taken is small, a detection subject region of alarger pixel size can be obtained by automatically performingzooming-in. Generally, when the number of pixel samples is small,digital watermark detection becomes difficult. Thus, by performingzooming-in to increase the number of the pixel samples in the detectionsubject region, performance for detecting digital watermark can beimproved.

As mentioned above, according to embodiments of the present invention,performance for detecting digital watermark is improved using variouscontrivances so that it becomes possible to take moving images on a TVand the like and perform digital watermark detection processes in realtime from the moving images.

In addition, a part in a captured frame image from which digitalwatermark detection should be performed is specified, so that digitalwatermark detection can be performed reliably under various conditionsof picture taking angles and background images, and that performance fordetecting digital watermark is improved. Accordingly, it becomespossible to take moving images on a TV and the like and perform digitalwatermark detection processes in real time from the taken moving images.

According to embodiments of the present invention, for example, viewingprogram related information can be obtained by taking images beingviewed on a TV with a camera-equipped mobile-phone, or by selecting achoice using watermark detection in a quiz program or in viewerquestionnaires and by sending the result to a broadcasting station bycommunication, interactive TV broadcasting can be realized in whichfeedback on program contents is obtained in real time. Or, in anaugmented reality system that requires barcode and the like in aconventional technology, the barcode can be unnecessary by usinginvisible digital watermark so as to solve unnaturalness, or, watermarkis used for game operation in a TV game by specifying an object that isan operation subject using watermark detection, or, applications thesame as those mentioned above are performed on an aurora vision on thestreet, or by embedding watermark into a display screen of a PC(personal computer) and by taking the image using a camera-equippedmobile-phone, information can be sent from the PC to the mobile-phone sothat liaison of apparatuses can be performed, or watermark embedding anddetection can be performed also for a still image on a display screen bycopying the still image to produce arbitrary n images (n is an integergreater than one), applying the present invention regarding the n imagesto be moving images of n frames, performing encoding using a method suchas animation GIF, and displaying the image on a browser screen and thelike. Or, when analyzing a hyper text such as one of HTML on a PC todisplay on a browser and the like, by embedding hyperlink informationinto a display region of an object having a hyper link, browsing isperformed with a camera-equipped mobile-phone by taking the displayscreen of the PC with the camera-equipped mobile-phone, or by performingthe above-mentioned application using a display screen of amobile-phone, linkage between mobile-phones and linkage from amobile-phone to a PC can be realized. As mentioned above, extremely widerange of application fields can be provided. Further, it is obvious thatthe present invention can be applicable to copy right management andcopy right protection that are usual application objects of digitalwatermark technology.

The invention claimed is:
 1. A digital watermark embedding method in adigital watermark embedding apparatus for embedding digital watermarkinto moving images, comprising: a step of inputting moving image dataincluding a frame image group, watermark information and watermarkpattern switching information specifying temporal change of watermarkpatterns; a frame image obtaining step of sequentially obtaining, byframe image obtaining means, each frame image of the moving image dataand frame display time that is display time of the frame image; awatermark pattern generation step of generating, by watermark patterngeneration means, a watermark pattern using the watermark information,the frame display time and watermark pattern switching information; awatermark pattern superimposing step of superimposing, by watermarkpattern superimposing means, the watermark pattern onto the frame image;and a moving image data reconstruction step of combining watermarkembedded frame images obtained by sequentially repeating processes ofthe frame image obtaining means, processes of the watermark patterngeneration means and processes of the watermark pattern superimposingmeans to generate watermark embedded moving image data.
 2. The digitalwatermark embedding method as claimed in claim 1, wherein, in thewatermark pattern generation step, timing for switching the watermarkpattern is repeated in a constant period.
 3. The digital watermarkembedding method as claimed in claim 1, the watermark pattern generationstep including: a step of generating the watermark pattern correspondingto the watermark information using the watermark information, the framedisplay time and the watermark pattern switching information; and stepsof generating a watermark pattern for phase determination, using theframe display time and the watermark pattern switching information, thatis used for estimating temporal change status of the watermark patternwhen performing detection, and multiplexing the watermark pattern forphase determination into the watermark pattern corresponding to thewatermark information so as to obtain a pattern as the watermarkpattern.
 4. The digital watermark embedding method as claimed in claim3, the watermark pattern generation step including: a step of dividingthe watermark information into data blocks using the watermarkinformation, the frame display time and the watermark pattern switchinginformation; and a step of generating the watermark pattern such thatthe data block ID also serves as watermark information for phasedetermination when generating the watermark pattern based on the datablock ID and data block information of the data block ID which aredetermined from the frame display time and the watermark patternswitching information.
 5. The digital watermark embedding method asclaimed in claim 1, the watermark pattern generation step including: astep of dividing the watermark information into data blocks using thewatermark information, the frame display time and the watermark patternswitching information; and a step of generating the watermark patternbased on a data block ID and data block information of the data block IDwhich are determined from the frame display time and the watermarkpattern switching information.
 6. The digital watermark embedding methodas claimed in claim 1, the watermark superimposing step including: astep of changing scale of the watermark pattern to a size equal to orless than the frame image so as to superimpose the watermark pattern inthe inside of the frame image.
 7. The digital watermark embedding methodas claimed in claim 6, the watermark pattern generation step including:a step of generating a basic watermark pattern using the watermarkinformation; a step of adding a positioning pattern for extracting adetection subject region when performing digital watermark detection tothe basic watermark pattern; and a step of generating the watermarkpattern by changing the basic watermark pattern using the frame displaytime and the watermark pattern switching information.
 8. The digitalwatermark embedding method as claimed in claim 6, the watermark patterngeneration step including steps of: modulating the watermark informationinto the basic watermark pattern using existing two-dimensional code;and generating the watermark pattern from the basic watermark patternusing the frame display time and the watermark pattern switchinginformation.
 9. The digital watermark embedding method as claimed inclaim 6, wherein the digital watermark embedding apparatus includes aplurality of watermark pattern generation means, and in the digitalwatermark embedding step, each of the watermark pattern generation meansgenerates a different watermark pattern, and the watermark patternsuperimposing means superimposes a plurality of watermark patterns ontothe frame image.
 10. The digital watermark embedding method as claimedin claim 1, the watermark pattern superimposing step including: a stepof amplifying amplitude of the watermark patter wherein the largermovement included in the frame image is, the greater the amplitude ofthe watermark patter for the frame image is.
 11. The digital watermarkembedding method as claimed in claim 10, the watermark patternsuperimposing step including steps of: storing a previously receivedframe image into storing means; and generating a difference imagebetween a currently received frame image and the previous frame image toamplify the amplitude of the basic watermark pattern based on pixelvalues of the difference image.
 12. The digital watermark embeddingmethod as claimed in claim 10, the watermark pattern superimposing stepincluding a step of amplifying amplitude of the whole of the watermarkpattern.
 13. The digital watermark embedding method as claimed in claim10, the watermark pattern superimposing step including a step ofamplifying a pixel region of the watermark pattern corresponding to apixel region where movement is large in the frame image.
 14. A digitalwatermark embedding method in a digital watermark embedding apparatusfor embedding digital watermark into moving images, comprising: a stepof inputting moving image data including a frame image group, watermarkinformation and watermark pattern switching information that is periodinformation specifying phase change of watermark patterns; a frame imageobtaining step of sequentially obtaining, by frame image obtainingmeans, each frame image of the moving image data and frame display timethat is display time of the frame image; a watermark pattern generationstep of generating, by watermark pattern generation means, a watermarkpattern using the watermark information, the frame display time andwatermark pattern switching information; a watermark patternsuperimposing step of superimposing, by watermark pattern superimposingmeans, the watermark pattern onto the frame image; and a moving imagedata reconstruction step of combining watermark embedded frame imagesobtained by sequentially repeating steps from the frame image obtainingstep to the watermark pattern superimposing step to generate watermarkembedded moving image data.
 15. The digital watermark embedding methodas claimed in claim 14, the watermark pattern generation step including:a basic watermark pattern generation step of generating a basicwatermark pattern using the watermark information; and a step of addingphase change determined in the basic watermark pattern to a nextprevious watermark pattern using the frame display time and thewatermark pattern switching information to generate a new watermarkpattern.
 16. The digital watermark embedding method as claimed in claim14, the watermark pattern generation step including: a basic watermarkpattern generation step of generating a basic watermark pattern usingthe watermark information; a sign pattern generation step of generatinga sign pattern based on pixel values of the basic watermark pattern; aphase change value calculation step of obtaining a watermark patternswitching phase change value corresponding to time difference from anext previous frame using the frame display time and the watermarkpattern switching information; watermark phase pattern generation stepsof providing signs of each element of the sign pattern generated in thesign pattern generation step to the watermark pattern switching phasechange value to obtain phase differences from a next previous watermarkphase pattern, and generating a watermark phase pattern for the currentframe using the phase differences; and a watermark phase pattern imagingstep of generating the watermark pattern based on the watermark phasepattern.
 17. The digital watermark embedding method as claimed in claim16, the watermark pattern generation step including: a step ofincreasing or decreasing amplitude of a corresponding pixel value in thewatermark pattern based on each pixel value of the basic watermarkpattern when generating the watermark pattern from the watermark phasepattern.
 18. The digital watermark embedding method as claimed in claim14, the watermark pattern generation step including: a step ofassociating phase difference change represented by the basic watermarkpattern with rotation amount in a coordinate system obtained from imagecomponents so as to generate the watermark pattern based on newcomponent values obtained by rotating by the phase difference change.19. The digital watermark embedding method as claimed in claim 18,wherein Cb-Cr components of an image is used as the image components.20. A digital watermark embedding apparatus for embedding digitalwatermark into moving images, comprising: means for inputting movingimage data including a frame image group, watermark information andwatermark pattern switching information specifying temporal change ofwatermark patterns; frame image obtaining means for sequentiallyobtaining each frame image of the moving image data and frame displaytime that is display time of the frame image; watermark patterngeneration means for generating a watermark pattern using the watermarkinformation, the frame display time and the watermark pattern switchinginformation; watermark pattern superimposing means for superimposing thewatermark pattern onto the frame image; and moving image datareconstruction means for combining watermark embedded frame imagesobtained by sequentially repeating processes of the frame imageobtaining means, processes of the watermark pattern generation means andprocesses of the watermark pattern superimposing means to generatewatermark embedded moving image data.
 21. The digital watermarkembedding apparatus as claimed in claim 20, wherein, in the watermarkpattern generation means, timing for switching the watermark pattern isrepeated in a constant period.
 22. The digital watermark embeddingapparatus as claimed in claim 20, wherein the watermark patterngeneration means: generates the watermark pattern corresponding to thewatermark information using the watermark information, the frame displaytime and the watermark pattern switching information; and generates awatermark pattern for phase determination, using the frame display timeand the watermark pattern switching information, that is used forestimating temporal change status of the watermark pattern whenperforming detection, and multiplexes the watermark pattern for phasedetermination into the watermark pattern corresponding to the watermarkinformation so as to obtain a pattern as the watermark pattern.
 23. Thedigital watermark embedding apparatus as claimed in claim 22, thewatermark pattern generation means including: means for dividing thewatermark information into data blocks using the watermark information,the frame display time and the watermark pattern switching information;and means for generating the watermark pattern such that the data blockID also serves as watermark information for phase determination whengenerating the watermark pattern based on the data block ID and datablock information of the data block ID which are determined from theframe display time and the watermark pattern switching information. 24.The digital watermark embedding apparatus as claimed in claim 20, thewatermark pattern generation means including: means for dividing thewatermark information into data blocks using the watermark information,the frame display time and the watermark pattern switching information;and means for generating the watermark pattern based on a data block IDand data block information of the data block ID which are determinedfrom the frame display time and the watermark pattern switchinginformation.
 25. The digital watermark embedding apparatus as claimed inclaim 20, wherein the watermark superimposing means changes scale of thewatermark pattern to a size equal to or less than the frame image so asto superimpose the watermark pattern in the inside of the frame image.26. The digital watermark embedding apparatus as claimed in claim 25,the watermark pattern generation means including: means for generating abasic watermark pattern using the watermark information; means foradding a positioning pattern for extracting a detection subject regionwhen performing digital watermark detection to the basic watermarkpattern; and means for generating the watermark pattern by changing thebasic watermark pattern using the frame display time and the watermarkpattern switching information.
 27. The digital watermark embeddingapparatus as claimed in claim 25, wherein the watermark patterngeneration means: modulates the watermark information into the basicwatermark pattern using existing two-dimensional code; and generates thewatermark pattern from the basic watermark pattern using the framedisplay time and the watermark pattern switching information.
 28. Thedigital watermark embedding apparatus as claimed in claim 25, whereinthe digital watermark embedding apparatus includes a plurality ofwatermark pattern generation means, and each of the watermark patterngeneration means generates a different watermark pattern, and thewatermark pattern superimposing means superimposes a plurality ofwatermark patterns onto the frame image.
 29. The digital watermarkembedding apparatus as claimed in claim 20, wherein the watermarkpattern superimposing means amplifies amplitude of the watermark patterwherein the larger movement included in the frame image is, the greaterthe amplitude of the watermark patter for the frame image is.
 30. Thedigital watermark embedding apparatus as claimed in claim 29, whereinthe watermark pattern superimposing means stores a previously receivedframe image into storing means, and generates a difference image betweena currently received frame image and the previous frame image to amplifythe amplitude of the basic watermark pattern based on pixel values ofthe difference image.
 31. The digital watermark embedding apparatus asclaimed in claim 29, wherein the watermark pattern superimposing meansamplifies amplitude of the whole of the watermark pattern.
 32. Thedigital watermark embedding apparatus as claimed in claim 29, whereinthe watermark pattern superimposing means amplifies a pixel region ofthe watermark pattern corresponding to a pixel region where movement islarge in the frame image.
 33. A digital watermark embedding apparatusfor embedding digital watermark into moving images, comprising: meansfor inputting moving image data including a frame image group, watermarkinformation and watermark pattern switching information that is periodinformation specifying phase change of watermark patterns; frame imageobtaining means for sequentially obtaining each frame image of themoving image data and frame display time that is display time of theframe image; watermark pattern generation means for generating awatermark pattern using the watermark information, the frame displaytime and watermark pattern switching information; watermark patternsuperimposing means for superimposing the watermark pattern onto theframe image; and moving image data reconstruction means for combiningwatermark embedded frame images obtained by sequentially repeatingprocesses of the frame image obtaining means, processes of the watermarkpattern generation means and processes of the watermark patternsuperimposing means to generate watermark embedded moving image data.34. The digital watermark embedding apparatus as claimed in claim 33,the watermark pattern generation means including: basic watermarkpattern generation means for generating a basic watermark pattern usingthe watermark information; and means for adding phase change determinedin the basic watermark pattern to a next previous watermark patternusing the frame display time and the watermark pattern switchinginformation to generate a new watermark pattern.
 35. The digitalwatermark embedding apparatus as claimed in claim 33, the watermarkpattern generation means including: basic watermark pattern generationmeans for generating a basic watermark pattern using the watermarkinformation; sign pattern generation means for generating a sign patternbased on pixel values of the basic watermark pattern; phase change valuecalculation means for obtaining a watermark pattern switching phasechange value corresponding to time difference from a next previous frameusing the frame display time and the watermark pattern switchinginformation; watermark phase pattern generation means for providingsigns of each element of the sign pattern generated in the sign patterngeneration means to the watermark pattern switching phase change valueto obtain phase differences from a next previous watermark phasepattern, and generating a watermark phase pattern for the current frameusing the phase differences; and watermark phase pattern imaging meansfor generating the watermark pattern based on the watermark phasepattern.
 36. The digital watermark embedding apparatus as claimed inclaim 35, wherein, when generating the watermark pattern from thewatermark phase pattern, the watermark pattern generation meansincreases or decreases amplitude of a corresponding pixel value in thewatermark pattern based on each pixel value of the basic watermarkpattern.
 37. The digital watermark embedding apparatus as claimed inclaim 33, wherein the watermark pattern generation means associatesphase difference change represented by the basic watermark pattern withrotation amount in a coordinate system obtained from image components soas to generate the watermark pattern based on new component valuesobtained by rotating by the phase difference change.
 38. The digitalwatermark embedding apparatus as claimed in claim 37, wherein Cb-Crcomponents of an image is used as the image components.
 39. Anon-transitory computer readable recording medium storing a digitalwatermark embedding program for causing a computer to function as adigital watermark embedding apparatus which comprises: means forinputting moving image data including a frame image group, watermarkinformation and watermark pattern switching information specifyingtemporal change of watermark patterns; frame image obtaining means forsequentially obtaining each frame image of the moving image data andframe display time that is display time of the frame image; watermarkpattern generation means for generating a watermark pattern using thewatermark information, the frame display time and the watermark patternswitching information; watermark pattern superimposing means forsuperimposing the watermark pattern onto the frame image; and movingimage data reconstruction means for combining watermark embedded frameimages obtained by sequentially repeating processes of the frame imageobtaining means, processes of the watermark pattern generation means andprocesses of the watermark pattern superimposing means to generatewatermark embedded moving image data.
 40. A non-transitory computerreadable recording medium storing a digital watermark embedding programfor causing a computer to function as a digital watermark embeddingapparatus which comprises: means for inputting moving image dataincluding a frame image group, watermark information and watermarkpattern switching information that is period information specifyingphase change of watermark patterns; frame image obtaining means forsequentially obtaining each frame image of the moving image data andframe display time that is display time of the frame image; watermarkpattern generation means for generating a watermark pattern using thewatermark information, the frame display time and watermark patternswitching information; watermark pattern superimposing means forsuperimposing the watermark pattern onto the frame image; and movingimage data reconstruction means for combining watermark embedded frameimages obtained by sequentially repeating processes of the frame imageobtaining means, processes of the watermark pattern generation means andprocesses of the watermark pattern superimposing means to generatewatermark embedded moving image data.